SPIKE: Kafka Streams App Learning

[EvanParish]

User Story - Business Need

User Story(ies)

As a VA Notify Engineer I want to know more about how a Kafka streams app works
So that we can build our integration when we are ready.

Additional Info and Resources

Check Jacob's notes on a simple Kafka streaming app.

There are various python packages available for reading events from a Kafka stream. We should look into each and decide on the best approach. The following are the ones currently being maintained.

• confluent-kafka - a package maintained by the developers of Kafka
• ~faust-streaming - a more up-to-date version of faust~

Engineering Checklist

Required

• [x] Write a test app to read data from a Kafka stream
  • [x] read data from an offset or timestamp
  • [x] deserialize data (AVRO)
  • [x] handle possible errors
  • [x] connect to a Kafka topic over SSL
• [x] Document understanding of how to connect to BIP's Kafka topic and read events from it

Note: This test app should be written outside of the repo somewhere. If you check Jacob's notes above, you can see some example code to get you started. He was running this code in the dev-test-instance EC2.

Acceptance Criteria

Required

• A test app has been created.
• We understand the main areas and concerns associated with connecting to a Kafka topic
  • How to connect via ssl (usually you just need to connect with the url of the kafka instance but when dealing with ssl, you need to provide the cert data)
  • How to start at different offsets (date or offset)
  • How to deserialize a stream (AVRO)
  • How to handle unexpected end of stream (sometimes you can run into an error like with serialization that shuts down the consumer, it's good to know how to handle that gracefully and/or recover)
• Successful connection to BIP Kafka topic
  1. able to read events from the topic (byte array)
  2. able to deserialize events from the topic

QA Considerations

For QA to populate. Leave blank if QA is not applicable on this ticket.

Potential Dependencies

Leave blank if n/a

Out of Scope

Leave blank if n/a

[mjones-oddball]

Hey team! Please add your planning poker estimate with Zenhub @cris-oddball @EvanParish @justaskdavidb2 @k-macmillan @kalbfled @ldraney @nikolai-efimov

[mjones-oddball]

Open question: Where would this POC code go?

• We are going to update the doc Jacob started: https://github.com/department-of-veterans-affairs/vanotify-team/blob/master/Architecture/kafka/kafka_libraries.md
• Actual code could be stored in Dev test instance with coordination from team

[nikolai-efimov]

Does anyone know if we'll be using a schema registry to share AVRO schemas or will it be part of the incoming message?

[cris-oddball]

@nikolai-efimov BIP is using the confluent.io schema registry for the topic (they own the topic). Since we aren't producing anything, we won't need to register any schemas. So, their producer is already sending events to the topic that are serialized with the Avros schema, we just need to de-serialize it. We also do not have access to confluent.io.

[nikolai-efimov]

@cris-oddball , we need the schema to deserialize the message. It can either be shared via a registry or it comes in as a header in a message. There are nuances in handling messages depending on which approach is used.

[cris-oddball]

@nikolai-efimov I hear you. They provided that schema to us, so I believe we can use it to de-serialize. I will try to get an example event off the topic for you to see if it comes in as a header.

[nikolai-efimov]

@EvanParish @mjones-oddball @tabinda-syed I'm pretty much done with the app and need help with two things to wrap this up:

1. Info BIP's Kafka topic (certs, keystores, schema sharing method (hard coded, registry, message header), server address, etc)
2. Test it in company's environment

[nikolai-efimov]

@cris-oddball , I've tested things with the schema you provided (thank you for that!), it all works well. Serialization and deserialization are tested. As is it makes it a hardcoded schema for now, however, Kafka provides ways of receiving updated versions of schema either via a registry or in the message.

[nikolai-efimov]

@EvanParish , @cris-oddball , I got the following reply from Thomas in slack. "Hey Nikolai. It's not just you guys. Every External Tenant I've been working with is seeing the same Error. I have already requested help in resolving the issue. We're cautioning against updating the Group ID setting until we've had a chance to look into this more, but in the mean time for verification the group that we have listed for VA Notify is "vanotify-dev".