Interface with VBMS/BGS Kafka: VRO to receive Claim/Contention events

[yoomlam]

Months ago, Yoom was approved access to the DEV env to connect to VBMS/BGS's Kafka for VRO to receive Claim/Contention events, which will eventually supplant notifications from va.gov or MAS.

• Ask Yoom to forward relevant emails
• Get the certs required to connect to their Kafka in DEV
• Test the connection
• Report on results
• Get an update on BAM/AIM's status on these Claim/Contention events (with Yoom's or Diana's help)
• Plan next steps with Diana or Yoom

[yoomlam]

Thread about LHDI and Kafka

[amprokop]

We have learned a bit more about this, but are still blocked on actually connecting. After a significant amount of back and forth, it seems to have been settled that an IRB submission is necessary to get our certificates refreshed.

The VANotify team used [kcat](https://docs.confluent.io/platform/current/app-development/kafkacat-usage.html#kcat-formerly-kafkacat-utility) to perform connectivity tests, which seems like a good path for us to take. That team considered confluent-kafka orkafka-python for an overall implementation. After some research, it seems that Java tooling is more advanced, but the contention classification team may wind up doing most of their work in Python.

Next steps are to make the IRB request official (Diana and I are joining a meeting in 45 minutes on this) and connect further with the VANotify team to discuss their work with BIE eventing.

[amprokop]

Just wanted to also pull out a potential networking hurdle mentioned in the above linked thread - I have not verified this, but folks in the thread think that Kafka connections from LHDI to BIE may be blocked by default.

if the connectivity is not 443/HTTPS then it may be blocked and require an ESECC request

[amprokop]

Notes from meeting with Stephen Quick and team:

Every event has its own Kafka topic, and every Kafka topic is versioned. Each topic will require more access.We will likely want to initially consume events from four Kafka topics:

• contention associated with claim
• contention classification
• contention deletion
• contention update

There is a fifth contention topic, contention completed- my guess is we would not consume that at first. The event topics do not contain all historical data from the beginning of time. The BIE events will aggregate info from AIM and corpDB.

They highly recommend kcat to test the connection.

Stephen mentioned a few steps in the process of connecting:

• Certificate exchange - set up as mTLS -Whitelist us using ACLs
• Connect with kcat

Stephen confirmed that the Kafka connection is not over port 443, we may need to make access requests to unblock our ability to connect from LHDI. I am thinking that this may also block our ability to test the connection, because I believe we would likely use a Lighthouse DI k8s sleep pod to do that?

Next week, the IRB will discuss and hopefully approve granting us access.

[lukey-luke]

connectivity was verified in #1159

@yoomlam

1. Can you please provide info on who to contact for "BAM/AIM's status on these Claim/Contention events "?
2. What are next steps? Pull contention messages from inside abd-vro?

[yoomlam]

@dianagriffin @pshahVA Can one of you help Luke? See his comment above.

[dianagriffin]

@lukey-luke cc/ @yoomlam Re: "BAM/AIM's status on these Claim/Contention events", that part of the original comment is no longer relevant because we know the events are in production now.

In terms of next steps, I'm hoping @amprokop can help identify next steps on the technical side. Can y'all help me understand what we learned about this from connectivity testing:

Stephen confirmed that the Kafka connection is not over port 443, we may need to make access requests to unblock our ability to connect from LHDI. I am thinking that this may also block our ability to test the connection, because I believe we would likely use a Lighthouse DI k8s sleep pod to do that?

I take it this did not block our connection test in #1159; is that because we only tested with kcat and haven't crossed this hurdle yet? Or did it turn out to be a non-issue?

[amprokop]

Luke set up the Lighthouse DI k8s sleep pod to run the test with kcat (correct me if I'm wrong!). I think this means that we LHDI does not block our ability to connect to BIE Kafka.

Re: next steps on the technical side— we've confirmed that we can connect to Kafka from LHDI in the dev8 environment, and that we are getting contention events.

In terms of technical next steps, I'd suggest a few:

• Make sure we can parse the data and that we understand the format
• Connect to a higher environment, maybe just prod (since we are only reading and not writing events, Stephen Quick suggested that it might be best to integrate against prod because it has the most realistic data)
• Spec out how to embed the BIE integration into VRO

@lukey-luke might have more and better ideas.

Overall, though, after we successfully connect to prod from an LHDI sleep pod, we've de-risked this BIE integration and we should have some confidence that we can connect to it as needed on our own timeframe, rather than waiting for other teams to support us. I think Luke might be better served shifting gears for awhile, so we can get the initial version of the contention classification service off the ground.

Thoughts?