search

keedio / flume-ftp-source

FTP network server is source of events for Apache-flume
80 stars 61 forks source link

saveMap(sizeFileList) on each element discover decreases performance #4

Closed lazaromedina closed 9 years ago

lazaromedina commented 9 years ago

Now only saves state on stoping main process: captura de pantalla 2015-01-01 a la s 21 37 08

commented for increasing speed discovering new elements captura de pantalla 2015-01-01 a la s 21 36 08

lazaromedina commented 9 years ago

There is no need on saving map of files on each loop. Only if ftp server is very slow and contains very huge amount of files (> 20000) the process of discovering this all new files may take long time and is necessary save them, and not wait until next pool. But as is related in comment above, saving this state before each pool decreases performance of general plugin, fast or reliable?, thats the question.

lazaromedina commented 9 years ago

saving map after putting new element or modifiyng value , just after image