search

NASA-PDS / nucleus

Nucleus is a software platform used to create workflows for the Planetary Data (PDS).
https://nasa-pds.github.io/nucleus
Apache License 2.0
0 stars 0 forks source link

Deploy Nucleus in Test env #89

Closed viviant100 closed 3 months ago

viviant100 commented 7 months ago

💡 Description

Deploy with the least level of performance/workers possible for nucleus in Test env.

⚔️ Parent Epic / Related Tickets

No response

tloubrieu-jpl commented 7 months ago

This tasks requires to rewrite the terraform scripts.

Some other parts of the of the deployment need to be terraformed (e.g. RDS database).

jordanpadams commented 5 months ago

status: working on improving terraform scripts to enable this deployment

tloubrieu-jpl commented 5 months ago

@ramesh-maddegoda is making good progress on that.

tloubrieu-jpl commented 5 months ago

The deployment works but the DAG still have permission issues.

jordanpadams commented 5 months ago

Status: Adjusting roles for access. Currently testing the DAGs with lowest privileges and fixing some of those issues. No blockers. Terraform has been improved. Plan to destroy/redeploy before ready for production.

tloubrieu-jpl commented 5 months ago

Ramesh tested the deployment from scratch : need to launch the terraform twice because credential expires but that is ok.

tloubrieu-jpl commented 4 months ago

The deployment fully work expect for harvest which needs a whitelisted IP to send data to the JPL AWS OpenSearch.

tloubrieu-jpl commented 4 months ago

For more flexibility in the future we might have multiple nucleus instance deployed in dev and test, using a prefix for example.

ramesh-maddegoda commented 4 months ago

Created pull request https://github.com/NASA-PDS/nucleus/pull/98 to update Terraform scripts, IAM policy files, Lambda functions and documentation to deploy PDS Nucleus in MCP Test and Prod environments.

tloubrieu-jpl commented 4 months ago

@ramesh-maddegoda need to update the terraform README and the PR is ready for review.

tloubrieu-jpl commented 3 months ago

Presentation done to the SA. Review went well. Ramesh will investigate using secret manager for harvest configuration.

jordanpadams commented 3 months ago

Status: Continuing to evolve and update terraform code.

ramesh-maddegoda commented 3 months ago

Verified that Nucleus can process CSS data from DUM. All the terraform related changes were merged in to the main branch with the PR: https://github.com/NASA-PDS/nucleus/pull/98

tloubrieu-jpl commented 3 months ago

Ramesh added SQS in the terraform script to queue the lambda calls. It is tested.