search

MetOffice / aws-earth-examples

Example code of how to freely use Met Office's weather datasets through Earth on AWS.
https://metoffice.gov.uk
Other
43 stars 15 forks source link

Cannot access data in MOGREPS-UK S3 bucket #2

Closed KatherineTomkins closed 5 years ago

KatherineTomkins commented 5 years ago

Hi, Believe it allows public access, I am able to list the bucket, but unable to retrieve files from it: aws-earth-mo-atmospheric-mogreps-uk-prd

Command: aws s3 cp s3://aws-earth-mo-atmospheric-mogreps-uk-prd/ 8732490bb7e5a710eea5ff25830aaad8a48c57cc.nc sample.nc Got: "fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden"

I know it is holiday season, appreciate your help!

Thanks, Yiqun

armeniopinto commented 5 years ago

I can confirm the same problem. It's definitely a permissions problem, @jacobtomlinson, is this something you can fix?

jacobtomlinson commented 5 years ago

Thanks for raising this. I don't seem to be able to reproduce this with my IAM user.

The bucket has public permissions, so perhaps the IAM user you are using also needs permissions.

My user account has full S3 read permissions. Perhaps your IAM user needs to be given permission to read from S3?

jacobtomlinson commented 5 years ago

Looks like this is an issue with S3. There seems to be a bug with applying the permissions correctly due to the way we are writing the files. I'm looking at implementing a fix now.

TheBadEye commented 5 years ago

As of right now I can confirm the following Illicits a: "errorMessage": "An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied"

Exception when the following code is executed in LAMDA: s3.Object("MY_TEST_BUCKET", "e6434c6f83a5af5fbd108beb048be7cb8335e9c6.nc").copy_from(CopySource = {'Bucket': "aws-earth-mo-atmospheric-ukv-prd", 'Key': "e6434c6f83a5af5fbd108beb048be7cb8335e9c6.nc"})

The target is confirmed as being there via SNS Topic QUEUE

UPDATE: you may need to add ""s3:GetObjectTagging", to your public user policy as it is needed when copying files from one destination to another

jacobtomlinson commented 5 years ago

Ah interesting. That also seems like a separate but valid issue.

jacobtomlinson commented 5 years ago

This is now resolved. Any new files created going forwards will have correct permissions. Apologies for the inconvenience.

armeniopinto commented 5 years ago

As of right now I can confirm the following Illicits a: "errorMessage": "An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied"

Hello @TheBadEye ! Everything should be working now, let us know if you're still having issues!