earthcube / scheduler

Scheduling approaches related to gleaner tooling
Apache License 2.0
0 stars 0 forks source link

gleaner/nabu service avoid/cleanup duplicates #127

Open valentinedwv opened 1 month ago

valentinedwv commented 1 month ago

If a service exists, we get an error. Cases:

  1. source is a long running process, so the schedule is short (usually because of testing)
  2. source finished, and container needs to be removed
  3. workflows reloaded, so services got lost. need to have some user interaction to clean them up

Ideas on how to handle these

  1. check if an existing process is running, eg #111. So send an slack for admin check
  2. if service is in a non-running state, remove it. see if a run is running, kill run
  3. at startup can we check for running workflows, and stop them?

we should add labels to services, this will make it easier to find them.

List services, include the status=true https://docs.docker.com/engine/api/v1.46/#tag/Service/operation/ServiceList look at service status, if there are running tasks, it's running, if there are competed tasks it is done if competed tasks =0 there is an issue.

https://docs.docker.com/engine/api/v1.46/#tag/Service/operation/ServiceDelete