Louisiana: ReportStream to Natus

[scleary1cs]

Story

As a LA (Natus) Lab, so that LA can receive NBS ETOR orders, I need a sample order to show up in my demo /staging LIMS environment.

Pre-conditions

• A demo staging LIMS environment exists.
• We have access to or have access to someone with access to a demo staging LIMS environment.
• We can format/send a baseline OML message.
• Additional stories will be created to make the rest of the order show up correctly.

Acceptance Criteria

• [x] Sample order can be parsed correctly and displayed on the demo LIMS.
• Create a document to outline technical onboarding details for the intermediary/ETOR based on what we learn from this integration. This does not need to be part of the ReportStream documentation at this time. (removing this from scoped based on conversation in Slack: https://www.slack.com/archives/C044P2CBEKS/p1724683357274229)

Tasks

• [x] Draft initial technical setup documentation
  • [x] Draft written @basiliskus @jcrichlake
  • [x] Technical review
• [x] Any follow-up stories have been created

Engineering

• [x] Get the SFTP site information
• [x] Receive example OML message from Natus
• [x] CDC to whitelist SFTP site (CDC ticket needed for this)
  • [x] Understand more about what it means to whitelist an SFTP site. What's the process.
• [x] Organization with sender and receiver created in RS
• [x] Onboard Natus engineer (Natus) to the organization
• [x] Send Natus Postman collections
• [x] Send Brad Coney details on the RS protocols. - @halprin
• [x] Send Natus sample OML while we wait for SFTP site whitelist. - @halprin
• [x] Sample OML successfully sent to Natus SFTP site
• [x] Get the Natus domain and URL endpoint.
• [x] Forward developer machine IP addresses to Natus engineer for whitelisting for Natus' ReST endpoints.
• [x] Confirm that the Postman collection, provided by Natus engineer, works.
• [x] Compare/contrast the Postman collection and what ReportStream expects in an HTTP receiver.
  • [x] Work with Natus engineer regarding setting up an Auth endpoint
  • [x] Notify Natus engineer that endpoint will need updated with specific ReportStream Receiver requirements
• [x] Update the la-phl.etor-nbs-orders receiver to use HTTPS. This probably involves Ott.
• [x] Payload encryption support built: https://github.com/CDCgov/prime-reportstream/issues/12925
• [x] Send test orders and see if they go through, or triage any failures.

Definition of Done

• [ ] Documentation tasks completed
  • [ ] Documentation and diagrams created or updated
  • [ ] Implementation guide (/ig folder)
  • [ ] ADRs (/adr folder)
  • [ ] Main README.md
  • [ ] Other READMEs in the repo
  • [ ] If applicable, update the ReportStream Setup section in README.md
  • [ ] Threat model updated
  • [ ] API documentation updated
  • [ ] Source code documentation created when the code is not self-documenting
• [ ] Code quality tasks completed
  • [ ] Logging added where useful
  • [ ] Code refactored for clarity and no design/technical debt
  • [ ] Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
  • [ ] Code is reviewed or developed by pair; 1 approval is needed but consider requiring an outside-the-pair reviewer
  • [ ] Code quality checks passed
• [ ] Security & Privacy tasks completed
  • [ ] Security & privacy gates passed
• [ ] Testing tasks completed
  • [ ] Load tests passed
  • [ ] Unit test coverage of our code >= 90%
• [ ] Build & Deploy tasks completed
  • [ ] Build process updated
  • [ ] API(s) are versioned
  • [ ] Feature toggles created and/or deleted. Document the feature toggle
  • [ ] Source code is merged to the main branch

Research Questions

Decisions

Notes

• CHG0032022 is the SFTP whitelist ticket.
• CHG0032169 is the new SFTP whitelist ticket.

[jcrichlake]

We ran the sample message we received from NATUS through our existing processes in the staging environment to see what our output was.

Potential Issues We Saw

• The SPM segment was almost completely blank, 2.2 is missing in our test_OML message and isn't filled in
• Within OBR segment there can be a value we can use for SPM 4
• PID6 is used for NK1.3 if it's missing, NK1.3 is missing. NK1.3.3 is also missing the name of coding system. Should have been something like HL70063
• ORC5 is receiving a label of unknown
• We add the timezone offset to our times
• ORC.12 (Ordering Provider) is disappearing from our original message This field is marked as RE
• OBX.11, O is converting into an unknown.
• OBX.29 and OBX.30 specify QST and AOE respectfully and we're dropping that.

Things to possibly look into

OBR is getting a P appended to OBR-11. This means pending specimen, we may not want this even though it's correct

Missing numeric data in OBX5 when given a decimal

[sfradkin]

Currently blocked waiting for the CDC to whitelist the SFTP site so that we can do further testing with Natus

[nvcllns]

Folks, is there an existing ticket link/number we can reference re: escalating to Jasmine? (a la " CDC to whitelist SFTP site (CDC ticket needed for this)") Cc @halprin

[halprin]

@nvcllns, I reached out to ReportStream to get a ticket number, but they were literally told today that the process has changed on how to whitelist the SFTP site. So, ReportStream is filing that new ticket now. Therefore, there's nothing to escalate to Jasmine currently. If ReportStream get's the run-around again with this newer process, I'll certainly get that ticket number.

[jcrichlake]

Private Zenhub Image

[jcrichlake]

Tagging the story as blocked, had a meeting on 11/30 with Natus engineer and Ott to get everyone aligned on how the authentication endpoint will function. Natus will update their auth endpoint to take a username and a public key from reportstream to authenticate against Natus. Natus engineer thinks this work may be done by middle of next week.

[jcrichlake]

I just finished my meeting with Natus engineer and Ott, the final piece for the ReportStream to Natus integration is for Report Stream to implement encryption at rest for the data that is being transferred to Natus. Currently it's encrypted in transit and when we send the data to Natus it gets decrypted by their server and sits in the directory in it's plain text format. Both sides agreed to start on this work and we are just waiting for the implementation to be done

[jcrichlake]

https://app.zenhub.com/workspaces/engagement-6166edbd409257001e09f1b2/issues/gh/cdcgov/prime-reportstream/12925

[nvcllns]

Clarifying question here - is the anticipated implementer here solely ReportStream engineers? Or are changes also required by Natus and/or Intermediary?

[jcrichlake]

From my understanding this is just work for the ReportStream engineers to make sure the message body is encrypted at rest once it end up inside of the Natus system

[jcrichlake]

Adding blocked label to this since Ott is working on Baptist Alabama items

[SaraleCDC]

Mo to sync Natus engineer tomorrow 2/28

[JohnNKing]

Follow up needed with Natus. This may be done now.

[JohnNKing]

@JohnNKing Revise story; scope to integration that supports message receipt by Natus; then create a new story for data visibility in LIMS

[JohnNKing]

An issue needed to be corrected with the receiver filter.

Test messages sent in the last week: 1 on Friday, 3 on Monday, 3 on Tuesday

Discussed during PM call yesterday. We're now awaiting confirmation of receipt from Natus.

[JohnNKing]

As of Aug 16: Corrected message was processed successful! Also, receipt by Natus was confirmed. (originally comment was added to #826 by mistake)