search

cognoma / task-service

Cognoma Background Tasks API
Other
3 stars 2 forks source link

Feature/mvp queue #6

Closed awm33 closed 7 years ago

awm33 commented 7 years ago

Motivation

Queue implementation for the Cognoma MVP.

API changes

Some fields have been removed for MVP or are not needed for this queue backing type.

Implementation Notes

It uses a postgres SKIP LOCKED based task table and should handle a decent amount of concurrency, even greater than the number of workers we have budgeted for (2 to 3).

Functional Tests

Testing against the API using curl and the beginnings of the ml-workers code.

awm33 commented 7 years ago

@dhimmel @cgreene @dcgoss @stephenshank Here is our task service MVP. Please take a look, even if it's just to help me catch Python issues.

awm33 commented 7 years ago

@cgreene I forget to do a search for print before creating the PR :). I'll remove those. I also need to update the README and docs

cgreene commented 7 years ago

@awm33 : Sounds good. Busy day so I probably won't have another chance to look at it. Approving based on those incoming changes addressing print & docs.

awm33 commented 7 years ago

Just realized I'm not updating the status to "failed_retrying", "failed", or "complete" based on updates from the worker. I need to add that logic and some tests for it.

I'm sure once it's being used by ml-workers, there will be more feedback / changes.

awm33 commented 7 years ago

@cgreene @dhimmel I just pushed up a small change to handle updating the status based on 'complete', 'failed_retrying', or 'failed' states along with some tests for that logic. If there's no feedback or feedback that requires changes with the next dayish, I'm going to merge it.

dhimmel commented 7 years ago

I just pushed up a small change to handle updating the status based on 'complete', 'failed_retrying', or 'failed' states along with some tests for that logic. If there's no feedback or feedback that requires changes with the next dayish, I'm going to merge it.

Sounds good.