Automate deployment of SDAP to MCP/EKS with Terraform

jjacob7734 commented 1 year ago

Automate deployment of SDAP to MCP/EKS with Terraform.

Acceptance Criteria:

[x] EKS Cluster provisioned in Unity-Venue-Dev
[x] Separate EKS node groups: (i) 1 m5.xlarge for SDAP orchestrator/webapp and 1 (scalable up to 3) r5.2xlarge (memory-optimized) for SDAP workers/executors
[ ] SDAP ingress component deployed
[ ] SDAP application load balancer deployed

jjacob7734 commented 5 months ago

Status Update

EKS cluster provisioning automated, including: (i) desired SDAP configuration of 1 orchestrator/webapp instance and 1 memory-optimized worker instance, and (ii) required Unity tags
Above cluster will NOT be persisted until development work is completed
Still need to configure required IAM role for S3 access (required for SDAP ingest)
Terraform code (work in progress) checked in here: https://github.com/unity-sds/unity-analytics-sdap/tree/main/terraform/terraform_sdap_on_mcp_eks

jjacob7734 commented 5 months ago

Status Update

Created a new deployment script to automate SDAP deployment in Unity-Venue-Dev.
Currently deploying in 2 steps: (i) cluster provisioning with Terraform, (ii) SDAP deployment. Can be reduced to 1 step (all Terraform) deployment once an issue is resolved that is blocking access to the EKS cluster from an EC2 bastion host in the same subnet.
Another issue encountered is that the Spark helm chart (a required prerequisite to be deployed before SDAP is deployed) is not found where it used to be and I am trying to find an alternative implementation that works.
All of the SDAP-related automation components are now part of this repo: https://github.com/unity-sds/unity-analytics-sdap

jjacob7734 commented 5 months ago

Status Update

I found that the Spark Operator helm chart is now managed by the Kubeflow organization (handed over to them from Google). I have a new deployment procedure for that now and after some days of debugging found the main change needed from what I had before is that I had to explicitly set webhook.port=443 to get it to work (got cryptic Java errors without that).
I updated my helm chart to use the latest working SDAP containers for the SDAP webapp, Solr, data collection manager and granule ingester.
I now have a scripted process to deploy Spark and SDAP on an EKS cluster in a venue. I deployed successfully to Unity-Venue-Dev.
I ingested 1 month of sample data from MUR 25 km SST that I had and tested a time series by port forwarding.
Next step is to expose the time series endpoint by: (i) deploy SDAP ingress component (nginx), (ii) provision SDAP application load balancer, (iii) configure DAPA/API Gateway
The work for deploying SDAP with CoG support is dependent on the automation being developed under this ticket, and will be tracked in https://github.com/orgs/unity-sds/projects/3/views/25?pane=issue&itemId=50941398. For CoG support in SDAP, I am tracking this PR and will try modifying my SDAP deployment to use alternative containers for the SDAP webapp, data collection manager and granule ingester built off of this PR's development branch: https://github.com/apache/incubator-sdap-nexus/pull/307

unity-sds / unity-analytics-sdap