search

skyplane-project / skyplane

🔥 Blazing fast bulk data transfers between any cloud 🔥
https://skyplane.org
Apache License 2.0
1.09k stars 62 forks source link

[S3] `MissingBucketException` for bucket accessible with AWS CLI but not owned by same account #510

Closed sarahwooders closed 2 years ago

sarahwooders commented 2 years ago

I'm able to access a bucket (in region 'us-east-1') and am trying to move it to another account which I have access to but do not down. However I get an exception saying the source bucket does not exist.

I suspect this is a bug because we rely on listing the account's buckets to check if the bucket exists - however a user may have access to a bucket they can copy from even if they are not the bucket owner: https://github.com/skyplane-project/skyplane/blob/main/skyplane/obj_store/s3_interface.py#L47. I run aws s3api list-buckets and the source bucket does not appear, which is probably why there is an error.

We should find a way to check if a bucket exists even if the bucket is not owned by the user - e.g. by listing bucket contents and seeing if that throws an error, or using an AWS CLI command.

narang-amit commented 2 years ago

Think get-bucket-acl might do the trick? Would you like me to take a stab at it or are you working on it already?

parasj commented 2 years ago

Hey @narang-amit, any progress on this issue? Happy to sync quickly about this over Slack!

narang-amit commented 2 years ago

Hey! Got started, been a bit busy with interviewing. Should be able to take a stab at it tomorrow. Will update.

parasj commented 2 years ago

Thanks. Note that https://github.com/skyplane-project/skyplane/issues/528 is now fixed, and I think these may be similar issues. The fix was to improve logic around checking for bucket existence.

sarahwooders commented 2 years ago

I get a different error on this transfer now, so I can add that and we can close this issue?