search

kdave / btrfs-progs

Development of userspace BTRFS tools
GNU General Public License v2.0
562 stars 245 forks source link

Feature Request: implement pthread #178

Open unquietwiki opened 5 years ago

unquietwiki commented 5 years ago

https://en.wikipedia.org/wiki/POSIX_Threads

Not entirely sure how to go about it (currently), but it might knock a few days off longer operations by being able to queue up several files at a time to be worked on for things like send, restore, and rescue.

kdave commented 5 years ago

Some of the operations are inherently sequential so the potential gains from parallelism are limited. This would be possible in some cases to split the parsing part and callbacks that eg. create files. Error handling of the asynchronous calls becomes harder.

In case of receive I don't see much potential gain, the parsing overhead is really small and the file operations are executed one after another. They touch the same filesystem so serialization still happens in kernel. There might be some gain, I can't tell for sure just by looking at the code. Prototype and analysis would be needed.

ghost commented 5 years ago

What about compression , crc32c and threading?

adam900710 commented 5 years ago

For btrfs-image we're already multi-threading, especially for compression.

For btrfs-check, no, please no. We're far from being fully functional and chasing limited performance gain can easily screw up performance.

IMHO, only compression/decompress can take advantage, while btrfs-progs doesn't really do compression/decompression except btrfs-image, so we won't get much improvement.

unquietwiki commented 5 years ago

@adam900710 @Gatak @kdave Based on the multi-TB recover process I've had running, the threading would be useful for the restore function; especially if it's waiting for certain files to decompress & de-duplicate, or restoring multiple smaller files. If there was a way to have btrfs-progs manage the compression level of files en masse, as you suggested; then threading would be good there as well. I haven't used the send or receive functions; that might benefit alongside how recover works?