Limit task count to 1 for ECS provenance cluster

[sjoshi-jpl]

Research options to limit task count to max 1 for ECS provenance cluster. The requirement is to not kick off another task until the existing task completes.

If a task runs for too long (goes over a certain time period), it should through an error and terminate.

Investigate possible solutions :

1. Capacity provider for ECS cluster using AutoScaling groups (will need SA support)
2. Update provenance script to cap # of documents per task

[nutjob4life]

Sagar reports: focusing on this right now

[sjoshi-jpl]

@tloubrieu-jpl - I do not have access to create an ASG so I've opened a ticket with SA team.

https://itsd-jira.jpl.nasa.gov/servicedesk/customer/portal/16/DSIO-4247

[sjoshi-jpl]

@tloubrieu-jpl @jordanpadams as discussed today, we might no longer need this ticket (or we can re-purpose it) since we don't need the ASG and instead we're going to use the lock-file approach with Lambda that SA team is working on using a separate ticket.

DSIO-4247

[sjoshi-jpl]

@tloubrieu-jpl is this where we're tracking lambda creation for ECS tasks? Or should that be tracked in #30 ?

[tloubrieu-jpl]

I don't know @sjoshi-jpl , maybe the lambda would make more sense in this ticket since it is orchestrating the provenance processes and guarantee that only one is running on each domain at once and especially that not a new porcess is launched if one already runs.

The monitoring is more an after the fact, metrics measurements.

[sjoshi-jpl]

Per latest update from Susan, they're working on creating the lambda as well the registry-sweeper cluster.

[sjoshi-jpl]

60 New ticket created for tracking this task post multi-tenancy. @tloubrieu-jpl we can close this one.