search

bbfsdev / bbfs

Big brother file system (distributed file system)
14 stars 5 forks source link

Interface of indexing is complex. Concider simplify it #155

Closed yarondbb closed 11 years ago

yarondbb commented 11 years ago

Using of Patterns is pretty basic. Concider remove use of pattern object

genadyp commented 11 years ago

one of the requirements was omitting certain kind of files from indexing. is it still relevant?

bbfsdev commented 11 years ago

Don't know, maybe we should have some "blacklist" of files which should not be indexed, @vshiray what do you think?

yarondbb commented 11 years ago

I found that the usage today is very time consuming: Each file which is indexex is taking ~55 mS (actual SHAI takes ~1.5 mS) More over.. when conetn data is getting bigger, each index is taking more and more time: (100mS after one minute) (200mS after one minute) ... Look at the log after few minutes: [INFO] [2013-06-02 13:49:12] [index_agent.rb:76:Yaron:start index 170] [INFO] [2013-06-02 13:49:14] [index_agent.rb:76:Yaron:start index 719] [INFO] [2013-06-02 13:49:14] [index_agent.rb:76:Yaron:start index 871] [INFO] [2013-06-02 13:49:15] [index_agent.rb:76:Yaron:start index 20]

Only 2 files where indexed in 1S

This is due to the Pattern\Indexer mutual work for each file

yarondbb commented 11 years ago

Also treat: hostname is taking ~30mS (doing it for each indexing)