Discovery: Observability for events in Production

[robrap]

Document the various needs for observability, and make recommendations and ticket what is needed immediately, and what can be deferred.

Some earlier notes on observability:

• Monitoring:
  • Are consumer pods up?
  • Is the queue draining?
  • Current processing delay?
  • Do we need to know when a pod dies?
  • Are we getting Kafka errors when consuming events? What kinds? (KafkaError has various error codes.)

From https://www.infoworld.com/article/3269207/busting-event-driven-myths.html:

“It is also important to have strong monitoring and observability solutions in place. You need to know which service sent which events, and who is subscribed to these events. Having good visibility into the flow of events will let you understand the system and troubleshoot it with more confidence and less guessing.”

• How would this work?
• What details should we capture? Presumably we need an event id.

Also note, although we don't want to fully implement CloudEvents, any fields required for observability that are defined in the OEP for CloudEvents should be used where possible. See https://github.com/openedx/openedx-events/issues/77. This may include an event id, for example, although I'm not clear on whether or not Kafka also includes an event id.

[robrap]

I asked something like the following of both Confluent and New Relic, and will post answers later.

I’m wondering about monitoring our Confluent Cloud usage in New Relic.

• I found https://discuss.newrelic.com/t/confluent-cloud-metrics/151643/10, but hoping there is something more detailed for getting Confluent Cloud Metrics API data to New Relic.
• I found https://www.confluent.io/blog/kafka-connect-for-new-relic/, but I’m guessing that is not what I need at this time. I’m not yet looking to get our actual event data into New Relic. Although, I’ve read about the importance of having observability that enables you to track the flow of a unique event across services, and not sure exactly how people generally accomplish this. I welcome any thoughts or links on this topic.

[robrap]

• New Relic dashboard with some info: https://onenr.io/0VRV6V5dvRa
• Are we really getting latency of 5-6 seconds this often? Seems concerning. How would we validate New Relic data?
• What do we do about event id? Repeat of note above.

[robrap]

1. Added notes to https://2u-internal.atlassian.net/wiki/spaces/AT/pages/174555142/How+to+Use+the+Event+Bus+edX.org#Observability. Mostly using our runbook to provide additional details.
2. Separately, we are working on improving CloudEvent headers and getting them into error logs, at a minimum.

[robrap]

I'd like to add a set_custom_attributes call for the message id in the consumer. I asked if that could be done as part of the CloudEvents ticket here: https://github.com/openedx/openedx-events/issues/77#issuecomment-1332761300. If not, maybe add a new task. Either way, maybe this task could be closed.

[robrap]

Created the new task https://github.com/edx/edx-arch-experiments/issues/121 for new custom attributes. Closing this discovery.