NSF AWS S3 buckets access for collaborators

[saeed-moghimi-noaa]

Soroosh,

Would you please help Dan in terms of how to set S3 bucket credentials and share it with other limited number of collaborators collaborators. I think Tim could help with this issue.

Thanks, -Saeed

[saeed-moghimi-noaa]

Here is information shared by Greg for another possible person to reach out to for this topic:

National Oceanic and Atmospheric Administration Mail - Global Extratropical Surge and Tide Operational Forecast System (G-ESTOFS) Buckets.pdf

[josephzhang8]

Thx Saeed.

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Saeed Moghimi @.> Sent: Monday, June 14, 2021 4:39 PM To: saeed-moghimi-noaa/on-demand-post-processing @.> Cc: Y. Joseph Zhang @.>; Assign @.> Subject: Re: [saeed-moghimi-noaa/on-demand-post-processing] NSF AWS S3 buckets access to collaborators (#4)

[EXTERNAL to VIMS received message]

Here is information shared by Greg for another possible person to reach out to for this topic:

National Oceanic and Atmospheric Administration Mail - Global Extratropical Surge and Tide Operational Forecast System (G-ESTOFS) Buckets.pdfhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsaeed-moghimi-noaa%2Fon-demand-post-processing%2Ffiles%2F6651142%2FNational.Oceanic.and.Atmospheric.Administration.Mail.-.Global.Extratropical.Surge.and.Tide.Operational.Forecast.System.G-ESTOFS.Buckets.pdf&data=04%7C01%7Cyjzhang%40vims.edu%7C35939c2afa50474e4a0c08d92f746fcc%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C1%7C637592999412459705%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pEOfEhjwFgW8pgT%2FGOgn5j5SGSqHQWtq7C%2FQDcqlElI%3D&reserved=0

- You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsaeed-moghimi-noaa%2Fon-demand-post-processing%2Fissues%2F4%23issuecomment-860976179&data=04%7C01%7Cyjzhang%40vims.edu%7C35939c2afa50474e4a0c08d92f746fcc%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C1%7C637592999412469654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ELSsc91R1e7XItMu%2FQK5HmFSs9GL%2FRvvXXQQrxClG10%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFBKNZ4KHYDLAIOH264XEMDTSZSGDANCNFSM46V5QAKA&data=04%7C01%7Cyjzhang%40vims.edu%7C35939c2afa50474e4a0c08d92f746fcc%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C1%7C637592999412479622%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YTFFFQhFRYk1N1AhgYVMrC9n7iUDRZGhgSGr2UFTXGc%3D&reserved=0.

[SorooshMani-NOAA]

@saeed-moghimi-noaa, I'd be happy to provide any help that I can. What is the requirement?

[SorooshMani-NOAA]

@danishyo, I briefly discussed the requirements with @saeed-moghimi-noaa and based on my understanding S3 bucket needs to give access to people who don't necessarily have accounts on AWS. Because of this, there are two solutions:

1. Keep a private S3 and add a frontend public facing EC2 for serving/authenticating the contents of S3
2. Make S3 public (have a public IP), but give access only to certain IPs, e.g. institution level IP for NOS or VIMS, etc.

Does that make sense? Am I missing anything?

[SorooshMani-NOAA]

Accidentally closed!

[danishyo]

Thanks Soroosh

Both options are OK to me, the question is, I don't know how to quick setup this on AWS. Could you show me how to configure this on AWS web or aws-cli interface?

[SorooshMani-NOAA]

These are good resources to keep handy about what permissions to grant or how to set things up:

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#specifying-grantee-predefined-groups
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html
• https://aws.amazon.com/premiumsupport/knowledge-center/block-s3-traffic-vpc-ip/
• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html
• https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistS3AndCustomOrigins.html
• https://aws.amazon.com/blogs/compute/uploading-to-amazon-s3-directly-from-a-web-or-mobile-application/

Let me do some basic research into what needs to be done to safely share S3 with a list of predefined IPs. I think that's the easier approach, although it might be less secure.

[SorooshMani-NOAA]

@saeed-moghimi-noaa, @josephzhang8 do you know what are IPs for NOS and VIMS that need to be allowed to have access?

[SorooshMani-NOAA]

@danishyo, is it clear at this point how the S3 bucket needs to be access? Through API? Through browser? etc.

[danishyo]

I'm not sure how others want to get the data. My original thoughts are through aws-cli or https link. If we go with aws-cli, we will need to create credentials for other users. If we go with https link, is it possible to create a limited access website on aws bucket?

Soroosh Mani @.***> 於 2021年6月16日 週三 下午3:54寫道：

@danishyo https://github.com/danishyo, is it clear at this point how the S3 bucket needs to be access? Through API? Through browser? etc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/saeed-moghimi-noaa/on-demand-post-processing/issues/4#issuecomment-862669405, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM3S4J42RSEQHI2WWZYF3VLTTD6OBANCNFSM46V5QAKA .

[SorooshMani-NOAA]

I'm exploring the option of creating a web site to serve the data. But I'm not sure how easy it is. Do you have permission to create IAM user? If so then you can easily create username for anyone who needs access and they can access it through AWS CLI. If not then we need to go back to the two options discussed above. In general it is not recommended to use a public S3 bucket, but I guess if the access is restricted to VIMS and NOS/OCS IPs it might be OK. HTTPS link seems to be a bit more involved, but I'm still reading about it.

[danishyo]

I don't think so, when I click IAM on web console, the following error shows: User: arn:aws:iam::203416866386:user/hyu05 is not authorized to perform: iam:ListAccountAliases on resource:

Soroosh Mani @.***> 於 2021年6月16日 週三 下午4:12寫道：

I'm exploring the option of creating a web site to serve the data. But I'm not sure how easy it is. Do you have permission to create IAM user? If so then you can easily create username for anyone who needs access and they can access it through AWS CLI. If not then we need to go back to the two options discussed above. In general it is not recommended to use a public S3 bucket, but I guess if the access is restricted to VIMS and NOS/OCS IPs it might be OK. HTTPS link seems to be a bit more involved, but I'm still reading about it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/saeed-moghimi-noaa/on-demand-post-processing/issues/4#issuecomment-862680893, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM3S4JZYVPFQKMTDOJVIX5DTTEATTANCNFSM46V5QAKA .

[josephzhang8]

On VIMS side:

chinook,sciclone.wm.edu == 128.239.56.27

-Joseph

Y. Joseph Zhang Web: schism.wiki Office: 804 684 7466

From: Soroosh Mani @.> Sent: Wednesday, June 16, 2021 2:54 PM To: saeed-moghimi-noaa/on-demand-post-processing @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [saeed-moghimi-noaa/on-demand-post-processing] NSF AWS S3 buckets access for collaborators (#4)

[EXTERNAL to VIMS received message]

@saeed-moghimi-noaahttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsaeed-moghimi-noaa&data=04%7C01%7Cyjzhang%40vims.edu%7Ced6543e0971c4aeb723008d930f82068%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C0%7C637594664531805483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FknvzAGNkBDtDfx0kqKkW3iTo8HToVGqSAAnnxccGVE%3D&reserved=0, @josephzhang8https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjosephzhang8&data=04%7C01%7Cyjzhang%40vims.edu%7Ced6543e0971c4aeb723008d930f82068%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C0%7C637594664531805483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6SF4z%2FfwZJJglayxWvTMm8sVt6cPOt3QGPX54dXPsWw%3D&reserved=0 do you know what are IPs for NOS and VIMS that need to be allowed to have access?

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsaeed-moghimi-noaa%2Fon-demand-post-processing%2Fissues%2F4%23issuecomment-862629151&data=04%7C01%7Cyjzhang%40vims.edu%7Ced6543e0971c4aeb723008d930f82068%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C0%7C637594664531815439%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=S%2B2zqC25LwNRVXohhnvRl%2FH6oHaczez0VKZKEf1879Q%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFBKNZ5ORMTEKUGSNDPWWATTTDXNDANCNFSM46V5QAKA&data=04%7C01%7Cyjzhang%40vims.edu%7Ced6543e0971c4aeb723008d930f82068%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C0%7C637594664531815439%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ljdu%2F99eI%2BBdse%2FCqIuN3CT5g2F83QHPDNgYOybDvTk%3D&reserved=0.

[SorooshMani-NOAA]

OK @danishyo , no worries. We'll try the other way then. Do you have time to have a meeting tomorrow afternoon. I'd like us to test the public S3 with limited IP access approach.

Thank you @josephzhang8 for the IP. We'll try to set it up for this IP tomorrow and see how it works.

[danishyo]

Could we do it Friday? Tomorrow may not work for me.

Soroosh Mani @.***> 於 2021年6月16日 週三 下午4:28寫道：

OK @danishyo https://github.com/danishyo , no worries. We'll try the other way then. Do you have time to have a meeting tomorrow afternoon. I'd like us to test the public S3 with limited IP access approach.

Thank you @josephzhang8 https://github.com/josephzhang8 for the IP. We'll try to set it up for this IP tomorrow and see how it works.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/saeed-moghimi-noaa/on-demand-post-processing/issues/4#issuecomment-862699239, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM3S4J6PWFNXQKPXHVZCXELTTECPBANCNFSM46V5QAKA .

[SorooshMani-NOAA]

Let's discuss when to talk on email so that this thread doesn't get too long. My bad for asking your availability in the first place.

[SorooshMani-NOAA]

@danishyo, can you check if on NSF cloud you don't have any IAM access at all or you are able to create roles? One clean way for setting up, without the IP thing I suggested earlier is to just access S3 bucket from an EC2 instance where anyone with a username can sign in and use the S3 or just SCP things to the S3 from the EC2 as SCP gateway.

[SorooshMani-NOAA]

Another way I can think of to help set up the system is S3 File Gateway. I'll try to play with Gateway a little bit on my personal account to see how exactly it works.

[SorooshMani-NOAA]

Some more helpful links

• https://www.youtube.com/watch?v=tP5edaxBEEI
• https://docs.aws.amazon.com/filegateway/latest/files3/what-is-file-s3.html
• https://docs.aws.amazon.com/storagegateway/latest/userguide/StorageGatewayConcepts.html?icmpid=docs_hp-volume-dashboard#storage-gateway-cached-concepts

[SorooshMani-NOAA]

After checking the access @danishyo has on AWS we realized EC2 and IAM access is denied. These two services are needed to setup a server to serve S3 to users without AWS account. We decided to contact TACC support to see if they can give necessary access.

Another approach we are considering is Globus connector for S3. We'll ask TACC support about this one as well.

There's also a serverless approach to S3, but that'll take more time and probably needs more access to services compared to EC2 method we tried and failed due to restrictions on the AWS access.

[SorooshMani-NOAA]

Given that this AWS is hosted by TACC, would it be possible for them to actually mount it for us on their system? If so then anyone with TACC access can get access to the S3 as well.

[SorooshMani-NOAA]

@danishyo, @saeed-moghimi-noaa has suggested that we can start with only serving files on S3 without the need to upload (at least for now) this can easily be done by making S3 a static website. The following policy examples for S3 are then used to make objects available and restrict some access. To allow public access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::Bucket-Name/*"
            ]
        }
    ]
}

and to restrict access to select IPs (CAUTION: make sure your IP is included in the list, otherwise you'll lose access to this S3 even from the console and need to ask an admin to revert permissions for you):

{
  "Id": "SourceIP",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SourceIP",
      "Action": "s3:*",
      "Effect": "Deny",
      "Resource": [
        "arn:aws:s3:::DOC-EXAMPLE-BUCKET",
        "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
      ],
      "Condition": {
        "NotIpAddress": {
          "aws:SourceIp": [
            "11.11.11.11/32",
            "22.22.22.22/32"
          ]
        }
      },
      "Principal": "*"
    }
  ]
}

[SorooshMani-NOAA]

Also relevant are the following links:

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/HostingWebsiteOnS3Setup.html#step3-edit-block-public-access
• https://aws.amazon.com/premiumsupport/knowledge-center/block-s3-traffic-vpc-ip/
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteAccessPermissionsReqd.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html

To get the files:

• https://stackoverflow.com/questions/40247230/wget-public-s3-bucket
• https://github.com/aws/aws-cli/issues/904
• https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects.html

The second link provides a way to use aws cli without aws account. This is useful for public S3 buckets. In short one can use AWS CLI like below to sync local directory with the S3 (only download from S3, no upload at this point)

aws s3 --no-sign-request sync s3://BUCKETNAME .

[SorooshMani-NOAA]

To grant both read and write permission to public in ACLs of the bucket, one can use aws s3api put-bucket-acl

[SorooshMani-NOAA]

Also relevant to giving permission to your account without specifying IP:

• https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_multi-value-conditions.html#reference_policies_multi-value-conditions-examples
• https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html#Conditions_ARN
• https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html#Conditions_IPAddress
• https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html

[SorooshMani-NOAA]

To make the S3 available publicly for given IPs and accounts, in "Permissions" first remove the check for all "Block public access" check boxes, then use the following policy:

{
    "Version": "2012-10-17",
    "Id": "SourceIP",
    "Statement": [
        {
            "Sid": "SourceIP",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<BUCKET NAME>",
                "arn:aws:s3:::<BUCKET NAME>/*"
            ],
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": [
                        "<IP1>",
                        "<IP2>"
                    ]
                },
                "ArnNotEquals": {
                    "aws:PrincipalArn": "arn:aws:iam::<ACCOUNT ID>:user/<IAM USER  NAME>"
                }
            }
        },
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET NAME>",
                "arn:aws:s3:::<BUCKET NAME>/*"
            ]
        }
    ]
}

Note that there's no need to change ACLs when using the policy above. ACLs predate IAM policies. If instead of user ARN you want to give the whole account full access do:

                "StringNotEquals": {
                    "aws:PrincipalAccount": "<ACCOUNT ID>"

[SorooshMani-NOAA]

@saeed-moghimi-noaa, @josephzhang8, the s3 is set up for public Get and List access. The easiest way to list and download the files is with AWS CLI using one of the following command:

aws s3 ls s3://tacc-nos-icogs2d-pds --no-sign-request # for 2D bucket
aws s3 ls s3://tacc-nos-icogs3d-pds --no-sign-request # for 3D bucket

Right now these buckets are only accessible to

1. Account holder (Dan)
2. NOAA East VPN IP
3. Frontera IP
4. VIMS IP

If we want SCP or Globus access, then we need to see if we can set up EC2. We'll discuss this with TACC cloud team tomorrow.

[saeed-moghimi-noaa]

@SorooshMani-NOAA Thanks Soroosh, Would it be possible to have static html access to the bucket as well? -Saeed

[SorooshMani-NOAA]

Yes, but someone should generate the html manually every time something is updated as far as I understand. Right now you can access specific files using https://<bucket-name>.s3.<Region>.amazonaws.com/<file_address.ext> for example https://tacc-nos-icogs3d-pds.s3.us-east-1.amazonaws.com/20210531/schism_elev_2021053003.png

[SorooshMani-NOAA]

If we enable web hosting and add an index.html file then one can do: https://tacc-nos-icogs3d-pds.s3.us-east-1.amazonaws.com/ but still someone has to update the index file(s)

[SorooshMani-NOAA]

Links related to static website hosting:

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html