HumanCellAtlas / data-store

Design specs and prototypes for the HCA Data Storage System (DSS, "blue box")

https://dss.staging.data.humancellatlas.org/

Other

40 stars 6 forks source link

Alert on failed notifications to Lira, define a process for dealing with failed notifications #1579

Open mweiden opened 6 years ago

mweiden commented 6 years ago

Need

User story 1

As a Data Store developer or operator
I want to know when a notification to the analysis pipelines is dropped
so that I can triage and resolve the issue.

We also need a process for triage and resolution of notification failures.

Creating issues
Resubmission or resending the event

Approach

Use the dcp-monitoring system to alert
Define a process for dealing with dropped notification and document it in the runbook

kozbo commented 6 years ago

Can we call this done? What is left?

mweiden commented 6 years ago

@kozbo this does have my head on it, but was hoping to do a little "knowledge share" on how dcp-monitoring works, so showed @Bento007 how to do this. I will switch the assignee.

@Bento007 if you're still up for this, it should only take ~30min or so. We can pair on it over VC if you like.

xbrianh commented 6 years ago

S3 object creation events are not always generated.

A mechanism for discovering un-evented objects may be useful.