Move the HHSProtect receiver to receive HL7v2 RADX Mars format, instead of CSV

sliu1000 commented 1 year ago

Problem statement

Jim Duff received a request from HHS Protect to "Move the HHSProtect receiver to receive HL7v2 RADX Mars format, instead of CSV"

What you need to know

Specifically, it must follow this "RADx MARS" implementation guide: ( https://www.nibib.nih.gov/covid-19/radx-tech-program/mars/hl7v2-getting-started).

This schema has fields that are not in our current CSV schema for Protect. So by simply switching, we can add a lot of value.
We current receive data in this RADx MARS format from senders (I think Intrivo?). So this now we want to send the data in this format.

NOTES:

1) Closely related to this, our friends at NIH, Krishna and Andrew, are doing an evaluation of the current CSV data we are sending to HHS Protect (they have access to the data in Protect), to look for missing data or other anomalies.
2) This is also closely related to ticket #7411 to make sure we are only sending OTC data to Protect.
3) Most important requirement: The data to Protect absolutely must continue to be deidentified.

Jim has told them this may not happen for a while, but I would like to get it ticketed.

To do

[ ] Assign to Engineer

jimduff-usds commented 1 year ago

RADx Mars team has been kind enough to write up a clear requirements document for how we should validate and filter senders who are adhering to the RADx/Mars spec.

See attached:

https://app.zenhub.com/files/304423150/6efbae0f-cf6b-4a78-96fb-99235297901c/download

jimduff-usds commented 1 year ago

@PatriciaPerozoUSDS , @avnieldravid : Needs discussion about whether to do this in the old or the new pipeline.

avnieldravid commented 1 year ago

@jimduff-usds does anybody from the onboarding team need to be there as well?

brandonnava commented 1 year ago

@jimduff-usds Do we know if this is going to be for the old or new pipeline yet?

PatriciaPerozoUSDS commented 1 year ago

I think there are some open questions still to determine which pipeline to do this in. From a long term perspective I think it makes the most sense to wait to do this in the universal pipeline. That would mean we would have to onboard STLTs that want to receive OTC data to the new pipeline and then change the senders to the new pipeline as well. If we are under pressure to migrate this within the next 6 months then it will have to go in the old pipeline.

sliu1000 commented 1 year ago

FYI - I emailed Andrew Weitz from NIH asking for support on how APHL currently sends their HL7 file to HHS Protect and have not gotten a response yet.

sliu1000 commented 1 year ago

@brandonnava @PatriciaPerozoUSDS @MauriceReeves-usds @brick-green @LaFreda1

Update from HHS Protect/Palantir:

Great to hear that we can start bringing ReportStream HL7 messages into Protect alongside APHL data. I’m looping in some of the Palantir engineers who can help with execution, and who set up the initial APHL ingest.

As we plan out this work, are there any example messages that you could share along with a data dictionary that notes which fields would be sent via HL7? It would also be helpful if HL7 messages and CSV raw files could be kept in separate locations for the time being, until we have fully configured and tested the HL7 pipeline. Perhaps we can aim for a kickoff planning meeting once we have a bit more detail here so we can scope this out and ask any follow-up questions we might have.

Thanks all,

John

LaFreda1 commented 11 months ago

Closing due to focus shift.

CDCgov / prime-reportstream