CDCgov / prime-reportstream

ReportStream is a public intermediary tool for delivery of data between different parts of the healthcare ecosystem.
https://reportstream.cdc.gov
Creative Commons Zero v1.0 Universal
71 stars 40 forks source link

Split out universal pipeline topics from `full-elr` to allow for `etor` topic to point to UP. #8592

Closed jimduff-usds closed 1 year ago

jimduff-usds commented 1 year ago

User Story:

As ReportStream, I should stop requiring that senders and receivers using the Universal Pipeline have a topic of full-elr because the Universal Pipeline should support different types of healthcare data.

Description/Use Case

Currently, the UP is tied to a 'topic' called full-elr. This is inaccurate, because it handles much more than ELR (eg, ETOR). Basically, currently we're using 'topic' as a synonym for 'which pipeline to use'. See SettingsProvider.kt:

/**
 * A submission with topic FULL_ELR will be processed using the full ELR pipeline (fhir engine), submissions
 * from a sender with topic COVID_19 will be processed using the covid-19 pipeline.
 */
enum class Topic(val json_val: String) {
    @JsonProperty("full-elr")
    FULL_ELR("full-elr"),

    @JsonProperty("covid-19")
    COVID_19("covid-19"),

    @JsonProperty("monkeypox")
    MONKEYPOX("monkeypox"),

    @JsonProperty("CsvFileTests-topic")
    CSV_TESTS("CsvFileTests-topic"),

    @JsonProperty("test")
    TEST("test"),
}

Risks/Impacts/Considerations

Dev Notes

The proposal for this is outlined here. The proposal suggests retaining the full-elr topic, but also adding an etor topic. Those topics will both signify usage of the UP, however senders for one topic will only be able to send data to receivers of the same topic. That routing aspect of these topics will be handled in a separate ticket (#9495).

The proposed design does acknowledge the coupling between topics and pipelines. Even though the topic won't directly specify a pipeline, the topic will still be used to determine which pipeline is being used. This is okay, since we only test/implement a topic with a specific pipeline in mind, and shouldn't allow for topics and pipelines to be decoupled as they are currently structured.

For reference, for now, I'll leave Arnej's other original proposal. We determined that it shouldn't be necessary for now, but it still may have other benefits that make it worth doing in the future.

Arnej: I think using some other method would be ideal. The topic system should be used only for topics, if not deleted outright. I think we should handle the pipeline switch at an API level (separate endpoints). We would copy the api/reports endpoint and make it api/v2/reports and deprecate the old, now legacy pipeline only, endpoint once the time comes. We would have to determine who would be affected by this (I think just CA and ETOR right now?). Since its not a lot of customers, I say we do this ASAP before more onboard.

Acceptance Criteria

thetaurean commented 1 year ago

Hey team! Please add your planning poker estimate with Zenhub @thetaurean @victor-chaparro @JessicaWNava @jsutantio @jimmyfagan @TurkeyFried