search

ShokoAnime / ShokoServer

Repository for Shoko Server.
http://shokoanime.com/shoko-server/
MIT License
386 stars 75 forks source link

[feature request] Make filehash single threaded #1144

Closed tamodolo closed 1 month ago

tamodolo commented 1 month ago

Or at least let user select how many instances of filehash process is running at given time.

VERSION INFORMATION

Server Version: 4.2.2

DESCRIPTION

This is a request to speed up hash process making it reading one file per time on HDD storages. Making it multithreaded on fast SSDs may make sense but not on HDD as the head will move like crazy trying to fetch all files at once causing massive overhead due to time wasted while moving the HDD head around. This is a speed up setting to make reads sequential.

Also I think this is mandatory as keeping video files on SSD are a bit overkill...

STEPS TO REPRODUCE

drop a lot of files at monitoring folder.

da3dsoul commented 1 month ago

Already on daily. To do so, you'd adjust the queue limit in settings-server.json

da3dsoul commented 1 month ago

And on stable, it only does one at a time anyway

tamodolo commented 1 month ago

Just checked, it's reading 2 files at the same time on version 4.2.2.0 (2cd0a0f)

Speeds are about 22mb/s but goes up to above 150mb/s that is normal sequential reads after one of 2 files ends reading.

I also searched settings-server.json and didn't found any option to control that queue...

edit: found the option

image

No reason to keep above 1 as it'll kill mechanical HDD performance. And HDD itself...

da3dsoul commented 1 month ago

There are plenty of reasons. Most of our users don't use a single hard drive. A RAID has acceptable concurrent read speeds. Multiple terabyte SSDs are becoming more common, as well. By the time you need Shoko, concurrent hashing is rarely an issue