Discovery to propose what technology should be used for the community edition Data Lake

[e0d]

Several new pieces of technology will need to be chosen to fill existing analytics gaps in the Open edX ecosystem. These pieces will fall into the following functional categories:

• Transform: How Tracking Log events will be transformed into xAPI events
• Transport: How events will be moved from Open edX applications to the data lake
• Storage: How the transport layer will store and query xAPI events
• Display: How raw events will be displayed as useful information to users

Overarching goals that inform these technology selections are:

• Open source over closed source
• Flexible deployment over vendor lock-in (self-hosted options should exist, SaaS deployment is a nice to have, any layer should be able to be swapped out by operators if they have the need)
• Configuration over code (ideally we should not need to create or support any new projects as part of this initiative)
• Operator simplicity over scalability (deep specialized knowledge should not be required to operate these technologies at small to medium scale, schema management should be kept to a minimum)
• Community-first (we should favor technology created and sustained by our partners, when possible)

Acceptance conditions are ADRs for each technology category explaining options considered, a clear decision made, and why:

• [ ] Transform ADR
• [ ] Transport ADR
• [ ] Storage ADR
• [ ] Display ADR

[jmakowski1123]

close in favor of discovery epic M1