🐛 Donor Dashboar aggregator is stuck/not working

rosibaj commented 3 years ago

Describe the bug

https://platform.icgc-argo.org/submission/program/MUTO-INTL/dashboard

Should have indexed data in the Raw Reads column for 10 donors, has none. These were publish 06/23 06/24 at the lates
using the dcc index kickoff does not appear to change for this program

Steps To Reproduce

Steps to reproduce the behaviour:

https://platform.icgc-argo.org/submission/dcc/dashboard
Click on MUTO-INTL index button

Expected behaviour

Index the stuff. not break.

joneubank commented 3 years ago

There is a recurring issue going on here where the aggregator gets stuck in a loop processing song analysis messages from kafka, while those messages are resulting in request failures and looping in the DLQ no other kafka events get processed.

I believe this is related to changes we made to not filter analysis requests by status, and this change will be moved to prod next release. We should re-evaluate then if this behaviour continues.

Until our changes are deployed, semi-regular restarts (scale down then up) of the service helps clear the error state and then the system will listen to requests to re-index programs.

joneubank commented 3 years ago

Once MUTO-INTL did run we found it miscounted the number of sequenced donors. We traced this issue back to the firstPublishedDate property of the analyses not being populated by Maestro in the RDPC song search index.

icgc-argo / donor-submission-aggregator