search

natcap / urban-online-workflow

This repository hosts the beta implementation of the Urban Online ES Workflow. The project is intended to give urban planners the ability to create and assess scenarios using InVEST Urban models.
1 stars 5 forks source link

Job queue: first draft #28

Closed dcdenu4 closed 2 years ago

dcdenu4 commented 2 years ago

This PR introduces a simple job queue for the backend that uses ansycio.PriorityQueue. For now the queue definition and the worker live in the endpoints file directly. In the future we would probably separate these out, having a separate process handle the workers. But, hopefully this is a good step that allows us to connect the frontend and backend initially.

Note: I have not thought through all the things we'll need to handle with interacting with queue and DB, such as failures.

I would love some feedback on whether this is a good first step for handling jobs or if we should make some edits here in this PR with a better initial setup.

dcdenu4 commented 2 years ago

but will we be able to run multiple worker()s at once?

I think we talked about this on a call, but yes, multiple workers() can be run concurrently, create_task allows this.

dcdenu4 commented 2 years ago

@phargogh, I think this is ready to go!