skyplane-project / skyplane

🔥 Blazing fast bulk data transfers between any cloud 🔥
https://skyplane.org
Apache License 2.0
1.09k stars 62 forks source link

Avoid reading recursive objects if we don't transfer #543

Open ShishirPatil opened 2 years ago

ShishirPatil commented 2 years ago

A skyplane cp s3://source s3://dest where each bucket has sub-directories will result in the following error

❌ In order to transfer objects using a prefix, you must use the --recursive or
-r flag.
If you meant to transfer a single object, pass the full source object key.
Try running: skyplane cp s3://source s3://dest --recursive

But before this, all the objects are queried:

Querying objects in skyplanelibri-us-east-1 (found 30000 objects so far)

Querying objects is expensive and bills the user. We know if the --recursive flag was used or not. If not, we need to stop at the first instance of a object with prefix is found and notify the user.

parasj commented 2 years ago

Is this a still an issue after @abiswal2001's change in #527?

sarahwooders commented 2 years ago

bump on this? will close otherwise

ShishirPatil commented 2 years ago

Error still exists.