Closed BinaryDevotee closed 3 years ago
Hello Athila,
This is something I originally wanted to achieve, BUT fpart in its current version is tightly bound to fts(3) and depends a lot on its features/characteristics (mostly depth-first traversal, dir entry sorting, ...) ; switching to parallel FS crawling would probably not provide those abilities and require a big rewrite. It is no more in the scope right now.
Best regards,
Ganael.
Hi, Ganael,
Makes sense. If you ever want to tackle this, I will be more than happy to test. :)
For now, thank you for your reply.
Regards
You're welcome :)
Cheers,
Ganael.
Is it possible to add multi-thread support to fpart, similar to the functionality -n from fpsync?
A little bit of context on it: I am dealing with a file system that has a big amount of small files (1 TB of 7M+ files), and partitioning them with fpart takes about 63 minutes with the parameters I specified. Conversely, by running:
find /path/to/filesystem/ -mindepth 2 -maxdepth 2 -type d | parallel -j8 find {} -type f | split -dl 100000 - list
it takes 36 minutes to complete. Of course, the produced list is not sorted but this is unimportant for my use case. Ideally, I would like to be able to achieve the same results with fpart by leveraging more parallel jobs.
Can this be achieved?