Deploy SDAP with COG support to Unity-Venue-Dev account

unity-sds / unity-analytics-sdap

Resources related to Unity SDAP deployment.

Apache License 2.0

0 stars 1 forks source link

Deploy SDAP with COG support to Unity-Venue-Dev account #15

Open jjacob7734 opened 8 months ago

jjacob7734 commented 8 months ago

Deploy the development branch of SDAP that supports Cloud Optimized GeoTIFF (COG) as the data store to the Unity-Venue-Dev project account.

Acceptance Criteria:

[ ] SDAP with COG support deployed to Unity-Venue-Dev
[ ] Ingest test data into SDAP
[ ] SDAP time series endpoint tested with test data

jjacob7734 commented 8 months ago

The SDAP branch with CoG support is actually in this fork by Riley Kuttruff: https://github.com/RKuttruff/incubator-sdap-nexus/tree/SDAP-484-CoG. That branch is not thoroughly tested and has not been merged into the main SDAP branch. Riley's development branch includes support for:

SDAP's normal datastore based on Solr and Cassandra
CoG
Zarr

Riley has Docker images tagged with SDAP-484-CoG that can be used to deploy his variant of SDAP:

rileykkjpl/sdap-nexus-webapp
rileykkjpl/sdap-collection-manager
rileykkjpl/sdap-granule-ingester

GodwinShen commented 8 months ago

@jjacob7734 can we please get an update on this ticket?

GodwinShen commented 7 months ago

@jjacob7734 ping again, please provide an update on this ticket. It was originally planned for 24.1.2 and the end of 24.1.3 is quickly approaching.

GodwinShen commented 7 months ago

From slack chat with @jjacob7734:

The SDAP deployment isn't done yet. I did get the development fork of SDAP that has the CoG support we need. I need to try deploying it. That is a prerequisite to the SBG L2 ingest (#16)

GodwinShen commented 7 months ago

@jjacob7734 can we get a quick update on this ticket?

jjacob7734 commented 7 months ago

@jjacob7734 can we get a quick update on this ticket?

I am adapting our Terraform for process mapper deployment to provision my new cluster automatically. I can do a manual provisioning using U-CS tools instead so I can proceed.

jjacob7734 commented 7 months ago

Working with Galen (U-CS) on process for provisioning new Unity EKS cluster.

jjacob7734 commented 7 months ago

Status Update

Working on Terraform automation of the SDAP deployment to MCP/EKS (see https://github.com/orgs/unity-sds/projects/3/views/17?pane=issue&itemId=30130124 for details)

jjacob7734 commented 6 months ago

Can provision and destroy the EKS cluster for SDAP on demand in Unity-Venue-Dev.
Created terraform.tfvars to set up Kubernetes nodegroups in the same minimal configuration that I have tested on Unity-Dev. That configuration is 1 nodegroup with 1 m5.xlarge instance as orchestrator/webapp and 1 nodegroup with 1 memory-optimized r5.2xlarge instance (scalable up to 3).
Added required Unity tags to the terraform deployment.

Working on:

Provision a small EC2 bastion host with kubectl, helm, and eksctl installed to in order to script (fully automate) the remaining steps
Automate deployment of the SparkOperator (SDAP's backend parallelization platform is Apache Spark).
Fork the SDAP code to create a deployable version with my helm chart modifications required in order to get SDAP to deploy in my minimal configuration. That is a more orderly and controlled deployment than the default, and I use Kubernetes node taints and selectors to accomplish that.

jjacob7734 commented 6 months ago

Moving this issue to the backlog until this dependency is resolved: https://github.com/orgs/unity-sds/projects/3/views/17?pane=issue&itemId=30130124