CIROH-UA / NGIAB-CloudInfra

NextGen In A Box: NextGen Generation Water Modeling Framework for Community Release (Docker version)
https://docs.ciroh.org/docs/products/nextgeninaboxDocker/
11 stars 16 forks source link

BigQuery cloud request - BYU #150

Closed danames closed 1 month ago

danames commented 4 months ago

1. Requester Information: This should include the name and contact information of the person making the request. Dan Ames

2. Project Information: Provide a brief description of the project and its goals. This can help the infrastructure team understand the context and purpose of the requested resources. Please highlight how this project will be benefit from and/or provide benefit to other resources on the shared infrastructure. This is a request for the Google BigQuery NWM historical forecasts API. Google has recently operationalized code to move historical forecasts and ongoing forecasts into the Big Query tabular format. Our effort at BYU includes building an API to simplify access to the tables through a key-protected API. Our research on this effort is about done and we want to roll our API code over to CIROH to deploy keys to any CIROH members that need this access.

3. Project Description: If your project involves developing software or scripts, briefly describe the software you plan to develop. Just the API code that will need to be run in the Google Cloud under the CIROH account.

4. Resource Requirements: Specify the compute, storage, and network resources needed for the project. Be as specific as possible about the number of resources required, and any specific configurations or capabilities needed. This information will help the infrastructure team determine the appropriate resources to allocate. We don't know eactly how much resources will be needed since it will scale with use.

Options:

  1. Cloud Provider: AWS/Azure/GCP GCP

  2. Required Services in the Cloud: GCP Big Query and API tools

    List of AWS Services

    • EC2
    • S3 – public, private, requester pay, bucket name suggestion?
    • EBS (Amazon Elastic Block Store)
    • EFS
    • RDS
    • VPC (Virtual Private Cloud)
    • DynamoDB
    • ECS
    • EKS (Kubernetes Cluster)
    • Lambda
    • Others: please list

    List of Azure Services

    • Virtual Machines
    • Azure App Service
    • Azure Kubernetes Service (AKS)
    • Azure Functions
    • Azure Batch
    • Azure Blob Storage
    • Azure File Storage
    • Azure Machine Learning
    • Azure Key Vault
    • Other: please list

    List of GCP Services

    • Google Compute Engine
    • Google Kubernetes Engine (GKE)
    • Google Cloud Storage
    • Google VPC
    • Google IAM
    • Google BigQuery
    • Google Cloud Functions
    • Dataflow
    • Other: please list

5. Timeline: Indicate the expected timeline for the project and when the resources will be needed. This information can help the infrastructure team plan and allocate resources accordingly.

We are nearly ready to roll this code to CIROH. Goal would be to do so before the developer conference.

6. Security and Compliance Requirements: If there are any specific security or compliance requirements for the project, these should be clearly stated in the request. This will help ensure that the necessary security measures are in place for the project.

CIROH will need to be able to issue keys for users

7. Estimation: Include any cost estimation or requirements for the project. This will help the infrastructure team select the most cost-effective solutions for the project.

AWS Cost Calculator: https://calculator.aws/#/

Google Cloud Pricing Calculator: https://cloud.google.com/products/calculator

Our current costs in texting have been negligble. We anticipate users generating $300-$500 of BigQuery requests per month combined.

8. Approval: Indicate the necessary approval processes or sign-offs required for the request.''

arpita0911patel commented 4 months ago

@danames Thank you for creating this request. We are currently waiting to get our CIROH Google Account from UA OIT. As soon as we get it we will take this project first to deploy on that.

arpita0911patel commented 1 month ago

BigQueryAPI project is created under CIROH Google Cloud account and deployed NWM BigQuery API to that one.