search

SpaceNetChallenge / utilities

Packages intended to assist in the preprocessing of SpaceNet satellite imagery data corpus to a format that is consumable by machine learning algorithms.
Other
248 stars 97 forks source link

Permissions on the spacenet AWS S3 bucket #83

Closed nseeburn closed 5 years ago

nseeburn commented 5 years ago

I'm having trouble accessing data from the SpaceNet AWS public dataset. From previous similar issues there might be a permissions issue on the spacenet bucket.

I've verified that my access keyid, secret access key and region are configured correctly and was able to successfully list one of the LandSat open dataset buckets at s3://landsat-pds

I've attached the list command I use: aws s3 ls s3://spacenet-dataset/ --request-payer requester

And the error message that is returned: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

dlindenbaum commented 5 years ago

Hello,

Thank you for trying to access the SpaceNet dataset. No permissions should have changed recently with the dataset.

I just created a new account and verified that

aws s3 ls s3://spacenet-dataset/ works with it.

One thing to try. SpaceNet is no longer a requester-pays bucket therefor can you try it with out the request-payer tag?

Otherwise it may be a more complex problem.

Thank you

nseeburn commented 5 years ago

Thank you very much for getting back to me, I really appreciate it, especially as you must get this a lot. Just tried it without the request-payer tag, no luck. In the case that the bucket isn't requester-pays, from my understanding (which I'm happy to be wrong about) I think wget-ing something in the repo might also work, but returns a 403 error for me.

dlindenbaum commented 5 years ago

We have a special permission on the site that requires an AWS account to access the data. Therefore wget is blocked.

Can you check permissions on the account you are using and also send the results of your command if you attach the --debug command

aws s3 ls s3://spacenet-dataset/ --debug

Thank you

nseeburn commented 5 years ago

Hey, thanks again for getting back to me. Confirmed I've got AmazonS3FullAccess and I've attached the debug log below.

logfile_1.txt

dlindenbaum commented 5 years ago

Hello,

Thanks for the log file. The log file looks correct. The only thing I can see is that your aws-cli is a couple of generations old at this point. However I don't think that should effect capability.

Dave

nseeburn commented 5 years ago

I've just succeeded with my own personal instance. Looks like an issue on this end I wasn't aware of. I will proceed to interrogate an admin. Sorry to have bothered you, and thanks again for your help!

dlindenbaum commented 5 years ago

Great, I'm going to close the issue. If you have any additional information that could help others and can provide it please do.