search

anaconda / anaconda-project

Tool for encapsulating, running, and reproducing data science projects
https://anaconda-project.readthedocs.io/en/latest/
Other
217 stars 88 forks source link

[ENH] Allow commands to be classified into terminating/non-terminating categories #346

Open mcg1969 opened 2 years ago

mcg1969 commented 2 years ago

Someone who is using anaconda-project now extensively for jobs has encountered an issue where they have a project with multiple commands—one intended to run as a persistent deployment, and another is intended to run as a scheduled job.

If you launch the deployment accidentally as a Kubernetes scheduled job, and you also don't know to set your concurrencyPolicy for such jobs to Replace or Forbid, then you have the unfortunate situation that jobs pile up until you exhaust your cluster.

It occurs to me that, in the spirit of capturing as much knowledge about a project as possible in anaconda-project.yml, that we could have an optional field that can be added to each command that specifies whether or not it is expected to terminate. For example, we could have a boolean field terminates, or a field type with categories api, application, job, etc.

jlstevens commented 2 years ago

I like this proposal but I would suggest that terminates: false is the implicit default as deployments are typically (in my experience) flask, notebook, bokeh/panel or other server processes that keep running.

There are some existing examples of terminating process that I've encountered in the wild, namely testing and linting.

One last thought (though I am not yet convinced this is a good idea!): if supports_http_options: true is set, that does strongly suggest a server process...

The type (maybe process_class is a clearer name?) suggestion is also nice as it is more general though it is hard to know the possible enumeration options when looking at the yaml whereas as a boolean predicate value is trivial to specify.

jbednar commented 2 years ago

terminating_job=True?

jlstevens commented 2 years ago

I do quite like that as it makes the most important use case clear...

jbednar commented 2 years ago

anaconda-project cron doesn't exist, but presumably some external service that launches such a scheduled job could give a warning if it finds one without terminating_job=True declared, asking the user if they really do want to schedule a recurring run for something that might not ever terminate.