steveloughran / cloudstore

Hadoop utility jar for troubleshooting integration with cloud object stores
Apache License 2.0
33 stars 11 forks source link

Add a -bfs option to the dux command for breadth-first-treewalk #4

Closed HarshitGupta11 closed 1 year ago

HarshitGupta11 commented 1 year ago

The -bfs option for dux does a breadth-first treewalk of the target filesystem, scanning subdirectories in parallel.

This is faster for high-latency object stores which don't have an O(1) deep tree list command (which S3 does; this option will be slower there)