broadinstitute / genetic-prevalence-estimator

https://genie.broadinstitute.org/
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Handle worker requests that repeatedly fail / time out #66

Open nawatts opened 2 years ago

nawatts commented 2 years ago

Currently, worker requests are retried a few times before being sent to a separate "dead letter" Pub/Sub topic.

https://github.com/broadinstitute/aggregate-frequency-calculator/blob/fa55b0beda132977eba115e80bf907fdecd465a5/terraform/pubsub.tf#L66-L69

Those variant lists will always show up as queued or processing. Something should subscribe to the dead letter topic and mark the associated variant list as errored / timed out.