Automate deployment of SDAP to MCP/EKS with Terraform.
Acceptance Criteria:
[x] EKS Cluster provisioned in Unity-Venue-Dev
[x] Separate EKS node groups: (i) 1 m5.xlarge for SDAP orchestrator/webapp and 1 (scalable up to 3) r5.2xlarge (memory-optimized) for SDAP workers/executors
Created a new deployment script to automate SDAP deployment in Unity-Venue-Dev.
Currently deploying in 2 steps: (i) cluster provisioning with Terraform, (ii) SDAP deployment. Can be reduced to 1 step (all Terraform) deployment once an issue is resolved that is blocking access to the EKS cluster from an EC2 bastion host in the same subnet.
Another issue encountered is that the Spark helm chart (a required prerequisite to be deployed before SDAP is deployed) is not found where it used to be and I am trying to find an alternative implementation that works.
I found that the Spark Operator helm chart is now managed by the Kubeflow organization (handed over to them from Google). I have a new deployment procedure for that now and after some days of debugging found the main change needed from what I had before is that I had to explicitly set webhook.port=443 to get it to work (got cryptic Java errors without that).
I updated my helm chart to use the latest working SDAP containers for the SDAP webapp, Solr, data collection manager and granule ingester.
I now have a scripted process to deploy Spark and SDAP on an EKS cluster in a venue. I deployed successfully to Unity-Venue-Dev.
I ingested 1 month of sample data from MUR 25 km SST that I had and tested a time series by port forwarding.
Next step is to expose the time series endpoint by: (i) deploy SDAP ingress component (nginx), (ii) provision SDAP application load balancer, (iii) configure DAPA/API Gateway
Automate deployment of SDAP to MCP/EKS with Terraform.
Acceptance Criteria:
Unity-Venue-Dev