OWP HAND-FIM Data Bucket - UA

sagycohen commented 4 months ago

1. Requester Information: This should include the name and contact information of the person making the request.

PI's Full Name: Sagy Cohen
PI's Affiliated Institute: University of Alabama
PI's Affiliated Email Address: Sagy.cohen@ua.edu

2. Project Information: As part of a CIROH project, we wish to develop a 'FIM as a Service' and 'FIM in a Box' CIROH community services.

3. Project Description: This request is to host the OWP S3 Bucket which contains the pre-calculated HAND rasters and other input needed to run the OWP HAND-FIM.

4. Resource Requirements: At this stage, we need to copy the OWP S3 Bucket which is about 1.5TB.

Options:

Cloud Provider: AWS/Azure/GCP AWS S3
Required Services in the Cloud: Data storage for applications

List of AWS Services
- S3 – public; Same name and structure and the OWP bucket.

5. Timeline: As soon as you can.

6. Security and Compliance Requirements: No

7. Estimation: Not sure

8. Approval: No special requirement to my knowledge.

benlee0423 commented 4 months ago

@sagycohen Could you please let me know your preferred name for the S3 bucket?

sagycohen commented 3 months ago

Ben, I’m adding Anupal Baruah to the discussion. @Anupal @.***>, what is the name of the OWP S3 bucket? Does it make sense to match the name for the CIROH bucket? Sagy

From: Benjamin Lee @.> Date: Friday, July 19, 2024 at 18:40 To: CIROH-UA/NGIAB-CloudInfra @.> Cc: Sagy Cohen @.>, Mention @.> Subject: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209)

@sagycohenhttps://github.com/sagycohen Could you please let me know your preferred name for the S3 bucket?

— Reply to this email directly, view it on GitHubhttps://github.com/CIROH-UA/NGIAB-CloudInfra/issues/209#issuecomment-2240697696, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABMQNNSTVXXBREFVDJD6IE3ZNGPXJAVCNFSM6AAAAABLBKXHVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBQGY4TONRZGY. You are receiving this because you were mentioned.Message ID: @.***>

sagycohen commented 3 months ago

Hi All

This is the script that I used to list the contents of the HAND FIM. I think we can use different bucket names or we can stick to the same.

!aws s3 ls s3://noaa-nws-owp-fim/hand_fim/ --request-payer

Anupal Baruah Post Doctoral Fellow Surface Dynamics Modeling Lab The University of Alabama Email: @.*** Phone-+1-659-261-6505/+91-8721886129

From: Sagy Cohen @.> Sent: Monday, July 22, 2024 8:58 AM To: CIROH-UA/NGIAB-CloudInfra @.>; CIROH-UA/NGIAB-CloudInfra @.>; Anupal Baruah @.> Cc: Mention @.***> Subject: Re: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209)

Ben,

I’m adding Anupal Baruah to the discussion.

@Anupal @.***>, what is the name of the OWP S3 bucket? Does it make sense to match the name for the CIROH bucket?

Sagy

From: Benjamin Lee @.> Date: Friday, July 19, 2024 at 18:40 To: CIROH-UA/NGIAB-CloudInfra @.> Cc: Sagy Cohen @.>, Mention @.> Subject: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209)

@sagycohenhttps://github.com/sagycohen Could you please let me know your preferred name for the S3 bucket?

— Reply to this email directly, view it on GitHubhttps://github.com/CIROH-UA/NGIAB-CloudInfra/issues/209#issuecomment-2240697696, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABMQNNSTVXXBREFVDJD6IE3ZNGPXJAVCNFSM6AAAAABLBKXHVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBQGY4TONRZGY. You are receiving this because you were mentioned.Message ID: @.***>

sagycohen commented 3 months ago

Let’s keep the name (‘hand_fim’) – it may simplify script adaptation.

From: Anupal Baruah @.> Date: Monday, July 22, 2024 at 09:12 To: Sagy Cohen @.>, CIROH-UA/NGIAB-CloudInfra @.>, CIROH-UA/NGIAB-CloudInfra @.> Cc: Mention @.***> Subject: Re: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209) Hi All

This is the script that I used to list the contents of the HAND FIM. I think we can use different bucket names or we can stick to the same.

!aws s3 ls s3://noaa-nws-owp-fim/hand_fim/ --request-payer

Anupal Baruah Post Doctoral Fellow Surface Dynamics Modeling Lab The University of Alabama Email: @.*** Phone-+1-659-261-6505/+91-8721886129

From: Sagy Cohen @.> Sent: Monday, July 22, 2024 8:58 AM To: CIROH-UA/NGIAB-CloudInfra @.>; CIROH-UA/NGIAB-CloudInfra @.>; Anupal Baruah @.> Cc: Mention @.***> Subject: Re: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209)

Ben,

I’m adding Anupal Baruah to the discussion.

@Anupal @.***>, what is the name of the OWP S3 bucket? Does it make sense to match the name for the CIROH bucket?

Sagy

From: Benjamin Lee @.> Date: Friday, July 19, 2024 at 18:40 To: CIROH-UA/NGIAB-CloudInfra @.> Cc: Sagy Cohen @.>, Mention @.> Subject: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209)

@sagycohenhttps://github.com/sagycohen Could you please let me know your preferred name for the S3 bucket?

— Reply to this email directly, view it on GitHubhttps://github.com/CIROH-UA/NGIAB-CloudInfra/issues/209#issuecomment-2240697696, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABMQNNSTVXXBREFVDJD6IE3ZNGPXJAVCNFSM6AAAAABLBKXHVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBQGY4TONRZGY. You are receiving this because you were mentioned.Message ID: @.***>

benlee0423 commented 3 months ago

@sagycohen We can only give public read access if this bucket is open to public. For writing to this bucket, we have a user assigned and give permission to that user. Anupal, do you have aws account with us already?

sagycohen commented 3 months ago

We need people to be able to download the data. Only admin should be able to upload.

On Jul 22, 2024, at 12:30, Benjamin Lee @.***> wrote:

@sagycohenhttps://github.com/sagycohen We can only give public read access if this bucket is open to public. For writing to this bucket, we have a user assigned and give permission to that user. Anupal, do you have aws account with us already?

— Reply to this email directly, view it on GitHubhttps://github.com/CIROH-UA/NGIAB-CloudInfra/issues/209#issuecomment-2243465643, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABMQNNXXCJKQQ35OUMIHFF3ZNU6SVAVCNFSM6AAAAABLBKXHVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBTGQ3DKNRUGM. You are receiving this because you were mentioned.Message ID: @.***>

benlee0423 commented 3 months ago

Bucket is created. Bucket name is globally unique, and I couldn't give the same with original bucket. Name of the bucket: noaa-nws-owp-fim-hand-fim

sagycohen commented 3 months ago

Can you rename it to ‘owp-hand-fim’?

From: Benjamin Lee @.> Date: Monday, July 22, 2024 at 13:46 To: CIROH-UA/NGIAB-CloudInfra @.> Cc: Sagy Cohen @.>, Mention @.> Subject: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209)

Bucket is created. Bucket name is globally unique, and I couldn't give the same with original bucket. Name of the bucket: noaa-nws-owp-fim-hand-fim

— Reply to this email directly, view it on GitHubhttps://github.com/CIROH-UA/NGIAB-CloudInfra/issues/209#issuecomment-2243598086, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABMQNNRSDGUKKGHXP4P4Z3TZNVHR3AVCNFSM6AAAAABLBKXHVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBTGU4TQMBYGY. You are receiving this because you were mentioned.Message ID: @.***>

benlee0423 commented 3 months ago

owp-hand-fim bucket created.

arpita0911patel commented 3 months ago

Let's keep it ciroh-owp-hand-fim if that's okay.

sagycohen commented 3 months ago

Thanks

From: Benjamin Lee @.> Date: Monday, July 22, 2024 at 14:03 To: CIROH-UA/NGIAB-CloudInfra @.> Cc: Sagy Cohen @.>, Mention @.> Subject: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209)

owp-hand-fim bucket created.

— Reply to this email directly, view it on GitHubhttps://github.com/CIROH-UA/NGIAB-CloudInfra/issues/209#issuecomment-2243625183, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABMQNNU4EHBZT3UU2EMZUATZNVJPRAVCNFSM6AAAAABLBKXHVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBTGYZDKMJYGM. You are receiving this because you were mentioned.Message ID: @.***>

sagycohen commented 3 months ago

Yes

From: Arpita Patel @.> Date: Monday, July 22, 2024 at 14:10 To: CIROH-UA/NGIAB-CloudInfra @.> Cc: Sagy Cohen @.>, Mention @.> Subject: [EXTERNAL] Re: [CIROH-UA/NGIAB-CloudInfra] OWP HAND-FIM Data Bucket (Issue #209)

Let's keep it ciroh-owp-hand-fim if that's okay.

— Reply to this email directly, view it on GitHubhttps://github.com/CIROH-UA/NGIAB-CloudInfra/issues/209#issuecomment-2243636425, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABMQNNWKQXYN3RWBQL5ZVS3ZNVKJ5AVCNFSM6AAAAABLBKXHVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBTGYZTMNBSGU. You are receiving this because you were mentioned.Message ID: @.***>

benlee0423 commented 3 months ago

ciroh-owp-hand-fim created

arpita0911patel commented 3 months ago

Thank you @benlee0423 . Closing this ticket.

CIROH-UA / NGIAB-CloudInfra

OWP HAND-FIM Data Bucket - UA #209

List of AWS Services