Redis Message Bus Discovery

[bmtcril]

Ticket formerly titled: Spike: Investigate redis streams as event bus

It is desirable to have a simpler event bus solution than Kafka for local development purposes in Tutor, and for running small-to-medium scale production Open edX installations. As we already have a dependency on redis, several community members have suggested using that as another message bus backend.

Additionally there is a desire to investigate pushing Open edX events over the event bus as a performant, resilient, scalable alternative to Celery. This ticket involves doing a proof-of-concept of both pieces.

Research Tasks:

• [ ] Familiarize yourself with the existing Event Bus architecture and where APIs exist to extend to a redis implementation. This page has a ton of fresh, very important, information: https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus
• [ ] Familiarize yourself with redis streams, especially consumer groups: https://redis.io/docs/data-types/streams/
• [ ] Familiarize yourself with Python packages that implement streams, possibly simplifying some implementation. https://pypi.org/project/redis-streams/ and https://walrus.readthedocs.io/en/latest/streams.html are two possibilities
• [ ] Document your decisions on how to proceed, and which tools you intend to you (and why)

The rest of this ticket can be adapted or split up if decisions above change assumptions, or the work seems too much for one ticket!

Create a PoC redis stream event bus implementing only what is necessary, and hard coding as much as needed:

• [ ] Create a clone of event-backend-kafka as event-bus-redis, renaming what is necessary, removing anything Kafka specific
• [ ] Make it installable in a local Tutor edx-platform
• [ ] Allow an event to be sent into a redis stream using a Django Signal
• [ ] Consume an event and output the contents
• [ ] Document the work needed to do this, and add push your changes to your private event-bus-redis for review

Investigate and document:

• [ ] Is it desirable to have one place where tracking log events are captured and sent to one or more redis streams instead of (or in addition to) disk?
• [ ] What changes would be necessary in event-routing-backends to work in an event consumer instead of as a send-time transformer? In other words, how difficult would it be difficult to have the consumer pull events from the stream, transform them to xAPI, and send them to an LRS?
• [ ] Would it be better / easier to simply have event-routing-backends transform the event first, then send it to the message bus instead of trying to put the raw event on the stream?

[timmc-edx]

When you get to the point of starting to implement, I have a draft PR open https://github.com/openedx/openedx-events/pull/85 for making an abstraction layer in openedx-events. This should be sufficiently broad to cover both Kafka and Redis implementations and allow openedx-events to load an implementation based on a Django setting.

[bmtcril]

That's great to know, thanks @timmc-edx !

[jmbowman]

I just rediscovered the list of reasons why we didn't choose Redis Streams in the first place (from https://github.com/openedx/event-bus-kafka/blob/main/docs/decisions/0002-kafka-based-event-bus.rst#redis ):

• Can lose acked data, even if RAM backed up with an append-only file (AOF).
• Requires homegrown schema management.

These may not be blockers for some use cases, but I figured it wouldn't hurt to drop in a reminder of these just to be sure of that.

[bmtcril]

Thanks @jmbowman! I remember those issues popping up in the initial investigation, and we're about to embark on a funded spike to help drive to a decision on whether this is a good route for smaller operators to take. There are bound to be some tradeoffs, but I'll make sure the effort to work around these issues is explicitly covered.

[robrap]

[request] It would be great if this work could include moving the OEP https://github.com/openedx/open-edx-proposals/blob/master/oeps/architectural-decisions/oep-0052-arch-event-bus-architecture.rst from Provisional to Accepted. As part of that, it could be updated with any additional ADR links that make sense for the Redis/Abstraction work.

[bmtcril]

Discovery has been done on this, now it's in implementation over on #27