Consume BIP contention event streams

[dianagriffin]

This is an epic representing new feature development for VRO to consume contention events and make that data integration available to partner products in VRO.

User Stories

• As a partner team, I want to have the capability to subscribe to contention creation and contention classification events, so that I can parse and analyze that data in measuring the accuracy of my contention classification service.
• As a VRO product owner, I want to consume contention events (create, classify, update, delete), so that VRO has access to this data as a foundation for future feature development to make claims data/metrics more accessible to product teams

Requirements

Consume/subscribe to the following topics from the BIE service:

1. contention created, TST_CONTENTION_BIE_CONTENTION_ASSOCIATED_TO_CLAIM_V02
2. contention updated,TST_CONTENTION_BIE_CONTENTION_UPDATED_V02
3. contention classified,TST_CONTENTION_BIE_CONTENTION_CLASSIFIED_V02
4. contention deleted, TST_CONTENTION_BIE_CONTENTION_DELETED_V02

Persist the events published to these streams somewhere that is queryable. We are interested in the following data elements for each event:

• The event type, e.g. “Contention created”, “Contention updated”, etc.
• The event timestamp
• Benefit Claim ID
• Contention ID
• Diagnostic Code(s)
• Classification (if available)

If events include other fields, it is likely fine to persist them -- please discuss with product owner. Please don't persist PII. Note that contention name/text can contain PII.

We don’t want this persistence to require a change to our ATO, so it should not exceed the policy we had in place for the data we persisted for RRD (which I believe was 90 days but please look at code to confirm).

Acceptance Criteria

1. VRO is connected to the BIE service in all non-prod and prod environments
2. VRO listens for/consumes/subscribes to all 4 kafka topics listed above
3. VRO persists the event data in a queryable storage where the query results are presentable to product owners.

Tickets within this epic

• 1662

• 1674

• 1675

• 1676

• 1677

• 1678

• 1679

• 1680

• 1682

• 1683

Not included in this work

This initial work will lay a foundation for future feature development using and analyzing contention event stream data. Further implementation of features/logic that are triggered by the event stream data will be covered by future tickets/epics.

Note s about work

• BIE kafka documentation
• related tickets (#1159 and #646)
• Slack threads documenting @lukey-luke's efforts to test connectivity in dev via kcat

[yoomlam]

Documenting results of engineering discussion:

• Implement a domain-specific BIE Kafka client under the domain-cc folder. If VRO Team implements this, we need a knowledge transfer and documented software requirements. Otherwise, Team CC can implement this with minimal infrastructure/DevOps support from the VRO Team.
• Create a ticket to generalize the BIE Kafka client so that it can be used by other domains and move the platform BIE Kafka client to a root-level svc-bie-kafka folder. One possibility is that this platform service would listen on RabbitMQ for Kafka-subscription requests and send Kafka-event notifications via RabbitMQ to the subscriber.

This strategy would minimize the domain's dependence on the VRO platform to implement a general BIE Kafka client. This can potentially free up the VRO Team to enable focusing on setting up the infrastructure for connecting to non-prod and prod BIE Kafka, setting up a Slack connection for non-prod and prod, etc.

[yoomlam]

Review how Yoom would accomplish this

[engineer-plus-plus]

" to have the capability to subscribe to [some kid of] events" " to consume [some kid of] events"

Just to clarify, do these two quotes mean the same thing? If not, an explanation what the author means may make the story more clear. If they are same, editing the story to use consistent language would, IMHO, make it easier to understand.

[engineer-plus-plus]

So the point of this is to listen to some types of evens in Kafka and forward them to Slack channels? Forgive me if it's an obvious question, but why? Why to Slack? So that what? So that " Contention Classification team member . . . can parse and analyze that data in measuring the accuracy of my contention classification service."? Are they to do it by hand? As in a human will be doing data entry from Slack to some form?

[yoomlam]

@engineer-plus-plus They are not exactly the same; maybe they are "2 sides of the same coin"? Basically, something needs to subscribe/register-to-listen to those events, and something needs to consume/handle the events as they come in.

[yoomlam]

I'm providing the following update to help @dianagriffin create sprint tickets. Further details will be provided as a result of the tech spec from #1662.

New approach: Implement a general (non-BIE-specific) Kakfa mock and client (tasks 2, 3, and 4); then updated them to work with BIE's Kafka service (tasks 5, 6, 7, and 8).

While there are more tasks than my original thoughts, the amount of effort is less. The tasks are smaller in scope and have fewer task dependencies, allowing us to get more done earlier. For example, tasks 1, 2, 3, and 9 can be started in parallel.

Tasks

1. Answer: For each relevant BIE environment, how do we connect to BIE Kafka (or what is the URL and authentication)? Implicit in this is: What are "all non-prod and prod environments" -- I presume this is referring to BIE environments?
   • Ask Luke to demo connecting to BIE in each env -- Connect to BIE contention event streams in DEV8 env #1159
   • How does DEV8 env map to LHDI envs? What are other BIE envs?
   • Create a Kafka-client container image; Go to each respective LHDI envs and test connection to BIE
2. Implement a mock general (non-BIE) Kafka service for testing and dev
   • Use a Kafka container image with minimal modifications
   • Save the Dockerfile in mocks/mock-bie-kafka/src/docker/
3. Implement a platform (domain-independent) Kafka client to connect to the mock general Kafka service
   • Depends on 2 in order to complete (but can be started with no dependencies)
   • Create a new microservice in a new svc-bie-kafka Gradle subproject
   • It needs to interface with other VRO microservices via RabbitMQ
4. Implement example domain code that uses the platform BIE Kafka client and saves Kafka events to the DB
   • Depends on 2 and 3
   • Add a new Camel route to domain-xample/xample-workflows/
   • Create an integration test that creates a Kafka topic and publishes events to the topic
5. Update the mock general Kafka service to be like BIE's Kafka service (for testing and dev)
   • Depends on 2
   • Create Kafka topics exactly like BIE's, including authentication
   • To facilitate integration tests, create a trigger (e.g., an API call) for events to be published to the topics
6. Update the platform BIE Kafka client to connect to the mock BIE Kafka service
   • Depends on 3
   • Depends on 5 in order to complete
   • Authenticate to mock BIE's Kafka service; only this microservice holds the credentials and is responsible for authenticating
7. Update example domain code to save BIE-specific event details to the DB
   • Subscribe to BIE topics
   • Update integration tests
8. Update Helm configs and Vault secrets with BIE credentials, and deploy to LHDI and test in each relevant env - Depends on 6
9. Set up DataDog to show saved DB contents

graph TD

subgraph Kafka
    subgraph BIE[BIE env]
        bie-kafka[kafka-server]
    end

    subgraph test_env[Test env]
        mock-kafka[mock-kafka-server]
    end
end

kafka-client -->|subscribe| bie-kafka -.->|event| kafka-client

subgraph VRO
    subgraph svc-bie-kafka
        kafka-client -.-> kafkaEvent-notifier
    end

    subgraph xample-workflow
        subscribe(subscribe to kafka topic)
        event-handler(kafkaEvent-handler) -.-> saveToDB
    end

    subscribe --> subscribeQ
    subscribeQ[\subscribe Queue/] --> svc-bie-kafka
    kafkaEvent-notifier -.-> kafkaEventQ[\Queue for kafkaEvent/] -.-> event-handler
    saveToDB -.-> DB
    DB[("Platform\n(DB)")]
end

DB --> DataDog

style DB fill:#aea,stroke-width:4px
style svc-bie-kafka fill:#AAF,stroke-width:2px,stroke:#777
style xample-workflow fill:#FAA,stroke-width:2px,stroke:#777
style test_env fill:#AAA,stroke-width:2px,stroke:#777
style Kafka fill:#EEE,stroke-width:2px,stroke:#777