Configure SDAP deployment in Unity Marketplace in Unity-Venue-Dev

unity-sds / unity-analytics-sdap

Resources related to Unity SDAP deployment.

Apache License 2.0

0 stars 1 forks source link

Configure SDAP deployment in Unity Marketplace in Unity-Venue-Dev #23

Closed jjacob7734 closed 2 weeks ago

jjacob7734 commented 6 months ago

SCOPE DECREASE:

the deployment to dev does not need to be done through marketplace, all we want in the existing sprint is:

mark 1-step deployment via terraform to "todo"
Manually (multiple terraform steps) deploy to unity-venue-dev manually, document that procedure.
Integrate SDAP with the unity-venue-dev API Gateway. Document that procedure
Integrate unity-dev dapa-api (in unity-dev )with the venue api (unity-venue-dev). Document this procedure. This is essentially the collection mapping bit that @zx128 did.

GodwinShen commented 4 months ago

@jjacob7734 ping for status.

jjacob7734 commented 2 months ago

Status

Plan to do this in these steps (planned for 24.3.3 sprint due to vacation):

EKS cluster provisioning
Spark operator deployment
Partial SDAP deployment to mainly get the worker node(s) set up with Solr, Cassandra, and Zookeeper. In this stage the driver node runs the web app, but only one Spark executor is deployed (the minimum).
Functional SDAP deployment with driver node set up with the ingest components and Spark executors scaled up to 3 per worker node.
Ingress/NGINX and load balancer deployment
API Gateway integration

jjacob7734 commented 2 months ago

Status:

Redeploying in Unity-Venue-Dev using new streamlined processes and latest security updates.
New Terraform automation scripted using the Terraform Helm Provider.
New Terraform deployment of the Spark Operator (one of the main SDAP dependencies) using the Terraform Helm Provider.
Now working on improving the parameterization of the Terraform Helm deployments and proceeding with the Terraform deployment of SDAP itself.
Plan to finish this conversion to Terraform deployment and deploy to all three venues in 24.3.4 sprint.

jjacob7734 commented 1 month ago

Status:

Terraform deployment now includes the Spark-Operator and SDAP components of Solr, Cassandra, Zookeeper, Webapp, and Spark executors.
Multi-stage deployment (all automated in terraform) ensures optimal mapping of the above components to a driver node and 1 or more worker nodes.
Remaining to automate with Terraform: SDAP Ingress/Nginx, Load Balancer, API Gateway/VPC Link

jjacob7734 commented 1 month ago

Status:

Completed Terraform automation of SDAP Ingress/Nginx, Load Balancer, API Gateway/VPC Link.
Now have multiple terraform apply steps to deploy from scratch.
Working on streamlining it a bit.
Also need to "hook up" the flow from the shared API gateway into the Venue API.

jjacob7734 commented 2 weeks ago

At end of 24.3 PI, SDAP deployed in Unity-Venue-Dev and accessible through the API Gateway (with Cognito Token) at https://0gaazm5nq8.execute-api.us-west-2.amazonaws.com/dev/sdap/.... Closing this ticket and future work may be delayed to a future PI as needed.