search

rejetto / hfs2

web based file server
https://rejetto.com/hfs
GNU General Public License v3.0
601 stars 131 forks source link

[Question] Indexing of VFS content ? #37

Closed VegethB closed 1 year ago

VegethB commented 1 year ago

Greetings, At the moment I am facing a problem and I would like to understand if it is due to the possible absence of an index or cache. I have 98gb of 23,000 files in 3250 folders, when I search, it also takes 3-5 minutes before I get results. I think it could be due to a lack of indexes, is it possible to implement improvements from this point of view? Like some modification in the template to use. Or is it a code-side thing of the program? Or am I doing something wrong?

hfs2.4RC06

Specs: DL380 G6 | x2 cpu Xeon l5630; Windows Server 2019 x64 DC LTSC; 32GB DDR3 1033mhz; Files on HDD sata2 active (sata3 supported); OS un SSD "";

Screen: image

Thanks in advance

rejetto commented 1 year ago

this is quite inherent to how hfs2 works. i don't exclude a clever template could solve it, one that is not reloading from scratch at each change, similarly to what hfs3 does.

my suggestion is to try hfs3 at https://github.com/rejetto/hfs#introduction I paid a lot of attention to cases with many files, and yet no indexes are involved. LMK

VegethB commented 1 year ago

this is quite inherent to how hfs2 works. i don't exclude a clever template could solve it, one that is not reloading from scratch at each change, similarly to what hfs3 does.

my suggestion is to try hfs3 at https://github.com/rejetto/hfs#introduction I paid a lot of attention to cases with many files, and yet no indexes are involved. LMK

Ok, thanks for the answer. I've already tried HFS3 and it was slower videoShowcase (bearing in mind that the results were populated as they went) but at the moment there is a need to 'upload' to these folders (function that is yet to be implemented). The video is a quick example, the 1st is HFS2 and the 2nd is HFS3. Weirdly it was much slower (the HFS.exe software runs in an ssd). Now (HFS2) it was very fast, because I did the test with no one connected, but in use with multiple users, the search slows down a lot. The VFS structures look different but actually in HFS3 I decided to directly use the root folder which contains all the folders seen in the HFS2 VFS (I don't think it changes the search speed because the excluded folders have no more than 2 files inside them). However, my intention is definitely to switch to HFS3 when some features currently absent compared to HFS2 (such as uploading) will be implemented.

Thanks again.

rejetto commented 1 year ago

your video is very interesting, i had no idea hfs3 was slower. Would you make a test using the same vfs, that is with root in hfs2 to confirm the timing is roughly the same? don't need any video proof. if it's confirmed I will make some testing to see how I can improve that.

rejetto commented 1 year ago

the time difference is frankly strange. I suspect there may be some strange folders like "recycle bin" that hfs3 is searching too.

VegethB commented 1 year ago

your video is very interesting, i had no idea hfs3 was slower. Would you make a test using the same vfs, that is with root in hfs2 to confirm the timing is roughly the same? don't need any video proof. if it's confirmed I will make some testing to see how I can improve that.

vSc2

Exact same VFS (I didn't hide 'Varie' but the rest is the same as HFS2's VFS) and it looks the same as before. If you need some specific logs to export with hfs2 and hfs3 for comparisons I have no problem sharing (as long as there is no sensitive information written)

rejetto commented 1 year ago

Ok, I had some spare time and worked on the hfs3's slowness problem. Fixed and published new version (0.26.5)

Made a final test searching a name in an archive of 32000 files, resulting in 3100 entries. hfs 2.4rc7 took 8.6s hfs 3 (0.26.5) took 0.7s 🥇

deal? :D

p.s. both times are so low because benchmarks ran several times and the OS has cached directories, but the important thing is that both are done in the same conditions

VegethB commented 1 year ago

What a nice new year surprise... I have one too 😭😂...

here > https://github.com/rejetto/hfs/issues/84

Anyway, thanks again 👍

VegethB commented 1 year ago

Ok, I had some spare time and worked on the hfs3's slowness problem. Fixed and published new version (0.26.5)

Made a final test searching a name in an archive of 32000 files, resulting in 3100 entries. hfs 2.4rc7 took 8.6s hfs 3 (0.26.5) took 0.7s 🥇

deal? :D

p.s. both times are so low because benchmarks ran several times and the OS has cached directories, but the important thing is that both are done in the same conditions

Ok, aggiornato a hfs3 0.26.7... Ottimo! Una velocità più che soddisfacente, grazie e buon anno nuovo!

Ok, updated to hfs3 0.26.7... Great! A more than satisfactory speed, thank you and happy new year!