FeatureBaseDB / SlothAI

A simple document pipeline manager for AI. Runs Python on AppEngine.
https://ai.featurebase.com/
MIT License
3 stars 2 forks source link

mostly async the task/process endpoint and refactor retry logic #11

Closed jrbrinlee1 closed 1 year ago

jrbrinlee1 commented 1 year ago

This PR changes the behavior of the /tasks/process endpoint.

Prior to the PR, this endpoint responded to the client only after the data had been inserted into the database (or not). Now, the endpoint just checks a few things, then creates a thread to handle most of the processing. This will make it easier for the client to send async requests.

This PR doesn't contain the work, but eventually we'll have a dead letter queue that contains all the bad messages.

Additionally, this PR simplifies retry logic. In the task processor. This will make it easier to implement a dead letter queue.