distcp / treeverse-distcp

[rabernat]

These tools looks really cool

• https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html
• https://github.com/treeverse/treeverse-distcp

[charlesbluca]

Thanks for bringing these up! In particular, the sync example using Hadoop DistCp looks like it could be particularly useful, although there are a lot of limiting factors:

• Would still need to be implemented in treeverse-distcp
• The time/computational resources needed to generate/persist snapshots of the buckets
• According to Apache "most cloud providers don’t work well with sync at the moment"

I think that as it is, treeverse-distcp could be a useful tool for Pangeo Forge in making recipes to copy/move data across cloud providers, and understanding how to take something like this and apply it to GCP Dataflow or other cloud batch processing services could be useful in the long run.