Some processing for maps, etc. will need a system for effectively running long-running tasks to, e.g. set topology, import/export maps, and calculate derived parameters. Right now some of this is happening in Kubernetes cron jobs or external listeners, but this has several disadvantages:
Processing steps are defined in Kubernetes configuration instead of in application repositories or in external watchers that have to stay running
There is little introspection on tasks, ability to cancel stalled tasks, etc.
The first step to get to a better organization is to set up a task/worker infrastructure in the Kubernetes environment. Right now, this can be a bit of an empty shell in anticipation of building out more functionality. Architectural considerations:
Celery is probably fine unless there is a compelling reason to use another orchestrator
For now, can have a simple "hello world" task, or tasks to calculate rgeoms for imported maps
Examples of how to kick off tasks from e.g., the API or Macrostrat's CLI
Some processing for maps, etc. will need a system for effectively running long-running tasks to, e.g. set topology, import/export maps, and calculate derived parameters. Right now some of this is happening in Kubernetes cron jobs or external listeners, but this has several disadvantages:
The first step to get to a better organization is to set up a task/worker infrastructure in the Kubernetes environment. Right now, this can be a bit of an empty shell in anticipation of building out more functionality. Architectural considerations: