Added in logic to retry the entire message broker connection creation process to extractor, worker & vdb worker. (still needs to be added to hugging_face worker)
Added logic to worker to retry batches that have failed. NOTE that this adds the messages back to the embedding queue, which can cause delays if there are new messages coming in while old ones are being retried. We will likely need to implement a separate retry queue and a worker to move jobs from that queue back onto the original queue
Fixed a bug in /embed caused by the telemetry
Verification
Can see that the request to both /jobs and /embed succeeds
What
embedding
queue, which can cause delays if there are new messages coming in while old ones are being retried. We will likely need to implement a separate retry queue and a worker to move jobs from that queue back onto the original queue/embed
caused by the telemetryVerification
Can see that the request to both
/jobs
and/embed
succeedssee actions for test results