NASA-PDS / registry-sweepers

Scripts that run regularly on the registry database, to clean and consolidate information
Apache License 2.0
0 stars 1 forks source link

Registry-Sweeper ECS Enchancements (Post Multi-Tenancy) #60

Closed sjoshi-jpl closed 2 months ago

sjoshi-jpl commented 10 months ago

💡 Description

All changes will be in MCP. This was being tracked under #31

  1. Create everything described below in MCP in order to kick off a Lambda Function to manage registry-sweeper tasks.

lambda execution rights, sns publish, cloudwatch logs dynamodb read access run ecs tasks, stop ecs tasks publish to sns topic pds-provenance-alarm query, scan and get items from dynamodb tables starting with pds-registry-sweepers ecs task role that can create/read/write to dynamodb any table starting with pds-registry-sweepers-

Susan created all of this under DSIO # 4341 in JPL AWS, so she can provide more details.

  1. The above lambda function will be triggered using an EventBridge schedule that runs hourly.
  2. We will also need to re-create all the CloudWatch Alarms / Monitoring in MCP using Terraform.
tloubrieu-jpl commented 2 months ago

This ticket is about preventing registry-sweeper tasks to start before the previous tasks are done.

tloubrieu-jpl commented 2 months ago

Instead of doing this ad-hoc development we prefer to use the configuration of the ECS cluster as proposed in ticket https://github.com/NASA-PDS/planetary-data-cloud/issues/105