search

National-COVID-Cohort-Collaborative / Data-Ingestion-and-Harmonization

Data Ingestion and Harmonization
41 stars 12 forks source link

N3C Data Pipeline Monitoring & Alerts #41

Closed DaveraGabriel closed 3 years ago

DaveraGabriel commented 4 years ago

Per Amit V: (he wanted...) "To inform the stakeholders that there shall be "monitoring and alerts" capability for system and data events notification that are currently under development and scheduled to be rolled out for N3C use in the next 2-3 weeks"

IN the DI&H pipeline, what events will trigger alerts? Who (which groups) will receive each kind of alert? Site Personnel N3C Admin N3C Dashboard(s) N3C DI&H N3C Analytics ... What is the alert delivery mechanism? eMail Site directory "manifest" & related reports

DaveraGabriel commented 4 years ago

@LisaOKeefe1 is responsible for writing up site / client verbiage outlining the notification the sites will get re: their data in the ingestion process. Am tagger her here to maintain the connection as we work through deploying these functions

LisaOKeefe1 commented 4 years ago

@LisaOKeefe1 is responsible for writing up site / client verbiage outlining the notification the sites will get re: their data in the ingestion process. Am tagger her here to maintain the connection as we work through deploying these functions

Not officially responsible. I've just had fellow PMs and others ask and want to be helpful to all if possible.

wakibbe commented 4 years ago

There is both the N3C operational view and the site notification piece, plus a general dashboard view of the milestone counts through the process. As we start doing this periodically (and as the period changes), showing the process of the latest data ingestion cycle, as well as each cycle. This will let us see potential issues as we change the overall process, QA steps, conformance.

DaveraGabriel commented 4 years ago

thanks @wakibbe ... a detail we should display on any dashboard status is the version of the payload. Presumably site payloads will happen frequently. We have established we will persist the latest 3 payloads on the sFTP server in the site-specific directory. This could get tricky on a dashboard for you: providing status for up to 3 payloads. A UI design nit for you to ponder...

DaveraGabriel commented 3 years ago

@wakibbe A component of the migration plan to a Production version for the DI&H pipeline includes notification triggers and schemas. The most recent version of this plan / checklist is provided here N3C+Systems+-+monitoring+and+alerting+subscriptions (1).docx

hlehmann17 commented 3 years ago

(I wonder when it will be time for a new drawing of the workflow.)

H

From: Davera Gabriel notifications@github.com Reply-To: National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization reply@reply.github.com Date: Wednesday, July 15, 2020 at 12:19 PM To: National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization Data-Ingestion-and-Harmonization@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization] N3C Data Pipeline Monitoring & Alerts (#41)

  External Email - Use Caution

@wakibbehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwakibbe&data=02%7C01%7Clehmann%40jhmi.edu%7C9ce7dc9be497464f4af808d828daa255%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637304267661364868&sdata=faf0cxPgXpnk%2F8vegPQNRVloX8R%2Bj6Fns5SxhZz802g%3D&reserved=0 A component of the migration plan to a Production version for the DI&H pipeline includes notification triggers and schemas. The most recent version of this plan / checklist is provided here N3C+Systems+-+monitoring+and+alerting+subscriptions (1).docxhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNational-COVID-Cohort-Collaborative%2FData-Ingestion-and-Harmonization%2Ffiles%2F4926459%2FN3C%252BSystems%252B-%252Bmonitoring%252Band%252Balerting%252Bsubscriptions.1.docx&data=02%7C01%7Clehmann%40jhmi.edu%7C9ce7dc9be497464f4af808d828daa255%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637304267661374863&sdata=e1QrWC4pBgGJC8RuOnIPglUXv95%2F%2BUu9ebMaXAO60IA%3D&reserved=0

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNational-COVID-Cohort-Collaborative%2FData-Ingestion-and-Harmonization%2Fissues%2F41%23issuecomment-658859570&data=02%7C01%7Clehmann%40jhmi.edu%7C9ce7dc9be497464f4af808d828daa255%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637304267661384862&sdata=7AAoQj2xK1fJ3SVpVw0PzBf7yAqj9RbETAtWGthsVes%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAEBMMJSA7L4NJ6AW6JOGMCLR3XJDDANCNFSM4OADJRBA&data=02%7C01%7Clehmann%40jhmi.edu%7C9ce7dc9be497464f4af808d828daa255%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637304267661384862&sdata=KA1evqBe7dvYQQ2HN3JJzmnu79HBdKPbRo9aOY5QVsA%3D&reserved=0.