AWS / Azure Resources for Model Assimilation Testbed - Calgary

[DaveCasson]

Request for Cloud Resources

1. Requester Information

Name: David Casson
Title: PhD Candidate, University of Calgary
Email: dave.casson@ucalgary.ca

---

2. Project Information

Project Title:

• Developing and Benchmarking Data Assimilation Methods on a Standardized Testbed

Brief Description:

• A part of the project will focus on the implementation of a SUMMA modelling framework for ensemble data assimilation. The cloud resources will be used for data preprocessing, model simulation and analysis, as well as running data assimilation experiments.

Goals:

• Assess improvements in streamflow prediction that can be achieved through ensemble data assimilation methods of available snow information.

Benefit

• This project will provide a basis for simulation and evaluation in the cloud resources, and explore the how model creation and analysis can build on available datasets.

  3. Project Description

Software and Tools:

• SUMMA Modelling Framework Software
• Python custom scripting (using repositories developed by collaborators)
• Jupyter Notebooks and Snakemake for workflow management
• Use of NetCDF operators (NCO) and Climate Data Operators (CDO)
• Transfer of meteorological and geospatial data from HPC datastores

Workflow:
Development will progress from meteorological data processing, data assimilation, hydrological model simulation, to forecast verification.

---

4. Resource Requirements

General Requirements:

• 32 cores
• CPU processing
• Storage: 500 GB

Specific Capabilities:

• Linux environment
• Fortran compiler
• Python installation
• IDE (e.g., Visual Studio Code)

Cloud Provider Options: AWS, Azure, GCP
Preferred Providers: AWS or Azure
Required Services in the Cloud: Not known at this stage.

---

5. Timeline

Project Duration: November 2023 to August 2025
Note: An initial allocation will ideally grow as the simulations move to ensembles and larger geographic domains.

---

6. Security and Compliance Requirements

• No specific security or compliance requirements known.

---

7. Estimation

Cost Estimation: Cost estimates are speculative at this stage due to a lack of experience with these cloud resources. It is requested to start with a modest allocation for initial learning and interaction with the resources.

8. Approval

Approval Process: TBD

[arpita0911patel]

Based on your resource requirement, wanted to let you know that we have access to UA on-premise servers (linux), and we'd like to know if they would be a suitable fit for your specific use case? or is there any dependency on using cloud services for this project?

[DaveCasson]

Hi Arpita, thanks for following up. On-premise servers could also fit the use case well. There is a late stage development (operational running of the test-bed) that would be better suited to cloud services, but that could come at a later time. Could you direct me to the on-premise server access?

[arpita0911patel]

You're welcome, and thank you for considering on-premise servers as an option for now. We can certainly provide you with access to on-premise servers. I'll collaborate with UA OIT to arrange VPN access for you. Please provide me with the list of users who will need access to that virtual machine via Slack. As of now, we have "awi-amd-lg-03" reserved for you, equipped with 16 cores and 128 GB RAM. If your usage demands additional cores, we can expand the allocation for you, offering up to 32 cores.

[arpita0911patel]

Sent the VPN details to Dave by email. Dave, please let us know if you face any issues.

[arpita0911patel]

This morning, I spoke with Dave and he is looking for HPC resources to use as a test-bed for the Summa modelling framework. Therefore, I would like to assign this ticket to Trupesh, so that he can create an account for Dave on Pantharhei HPC.

[TrupeshKumarPatel]

Hello @DaveCasson, Once you have UA credentials to log into VPN, please email me for Pantarhei account creation. Ensure the email body includes your full name and UA email address.

[DaveCasson]

Hi @TrupeshKumarPatel,

I was able to access Pantarhei with the credentials provided.

Thanks also @arpita0911patel for the help!

This issue can be closed from my perspective, but will leave that for your procedure.

[arpita0911patel]

Okay thank you Dave for the update. Closing this ticket.