Learning Core Platform Arch Discovery

[ormsbee]

Extract core learning concepts and data models into a new openedx-learning repository, with goal of creating a new, scalable core platform for learning innovation.

Goals

1. Refactor the Open edX LMS to enable more dynamic behavior in a scalable way. New features like V2 Content Libraries and Effort Estimation need a more scalable implementation than what exists in courseware today.
2. Provide an easier, more reliable path for extensions developers. Extensions authors should be able to build and test against a smaller repo than edx-platform.
3. Accelerate monolith breakup. By delivering incremental improvements to extensibility and performance, we have more reason to work on monolith breakup than our previous "extract all the non-core things" approach, where the benefits were more back-loaded.
4. Improve separation of edX business concepts from the learning platform. No edX-specific logic would come over to the new repo.
5. Advance the decoupling of Studio and the LMS. Part of this work includes building a new core content data model for the LMS, which would replace the current shared model with Studio.
6. Create a lightweight foundation for entirely new learning experiences. This would allow groups like LabXchange to make use of helpful infrastructure and concepts, without having to simultaneously inherit all the technical debt and weight of edx-platform.

Major Components

The high level components would include:

Composition

What permutation of a single unit does a user see? This would handle things like A/B tests, randomized problem selection, disabling content by enrollment type, adding staff-only debug markup, etc. There should be multiple backends for what can render a Unit, with the XBlock runtime being one of those.

Navigation

Sequence and Unit metadata, outlines, etc. How do you get to a particular piece of content you need to learn from? This would pull in parts of the Learning Sequences API from edx-platform.

Partitioning

Low level utility that helps determine what users are in what groups for the purposes of A/B testing, enrollment tracks, etc. Used by both Composition and Navigation.

Policy

Content-related settings as they apply to the site as a whole, organizations, and individual courses. This covers a lot of what Course Overviews and course override waffle flags do today.

Publishing

Centralized list of Learning Contexts (e.g. Courses, Libraries), their published versions, and various content-related errors and warnings are associated with them. We need this to help us tackle the major issue we have today around publishing: it's an asynchronous process with many different components that sometimes take minutes to complete and may fail independently, leading to a mixed-published state. This is the most foundational component that others will be built on top of.

Scheduling

Lower level library for content scheduling information, likely pulling in most of what is edx-when today.

Discovery/Design Phases

• [x] https://github.com/openedx/openedx-learning/issues/1
• [x] https://github.com/openedx/openedx-learning/issues/4
• [x] https://github.com/openedx/openedx-learning/issues/5
• [ ] Create an MVP XBlock Unit renderer.

Full Implementation Phases

• [ ] Phase 1: Publishing and Policy Establish foundation and start experimenting with CourseOverview data commonly wanted by extensions.
• [ ] Phase 2: Partitioning and Composition This would be useful for both upcoming V2 Content Libraries and Effort Estimation work.
• [ ] Phase 3: Scheduling Much of this could be ported over from edx-when.
• [ ] Phase 4: Navigation _Much of this could be ported over from the learning_sequences API._

Implementation Strategies

The following are some high level approaches/considerations with this new project.

Focus on content first.

Content data that lives in Studio is easy to re-build and backfill into new apps. User data is up to five orders of magnitude larger, and involves much larger challenges in terms of data migration.

Build Extensible Primitives

LearningContexts are a generic term that applies to Courses, Content Libraries, Learning Pathways, and any number of other collections of content that we want to discretely version and publish. We can centrally define logic around these, while leaving it up to higher layers to model specific types of LearningContexts in a pluggable way.

For instance, it makes sense for there to be a table of Courses, that have a foreign key to the LearningContexts table, which holds course-specific metadata. That table would have course-specific fields, and may even have a null learning context in the beginning (before any content is created).

This kind of arrangement would lead to a three layered system:

1. Foundational primitives and logic, implemented only in openedx-learning.
2. Specific implementations of those components to define different learning experiences, e.g. "Two-level hierarchical navigation", "randomized user partitioning scheme". These could be implemented in openedx-learning or outside. The most popular and useful ones could get folded into the repo over time.
3. Business-level plugins that tie into systems defined in (2), such as EnrollmentTrackOutlineProcessor. These would be implemented outside the openedx-learning repo (many would live in edx-platform).

Implement Plugins in edx-platform

We once attempted to lift the ModuleStores out of edx-platform and ran into a rat's nest of dependencies that made the task extremely difficult. My thought with things like this is to have plugin interfaces that go the other way from what we usually do–where the core framework logic is in apps in this new repo, and the little plugin objects are created in edx-platform (and optionally elsewhere as well).

So to use a concrete example, say we migrate the learning_sequences app in edx-platform today to become part of the navigation app in this new repo. The navigation app will then have the concept of OutlineProcessors–an object interface for different concerns that have to modify the set of things you can see or access in a course outline. The navigation app would have the logic for running OutlineProcessors, reading those values from a list defined in Django settings. The EnrollmentOutlineProcessor would be defined in edx-platform (and likely have imports to a bunch of things also in edx-platform), and then be specified in the settings file.

By doing this, we can keep some of the crazy logic and dependencies in edx-platform–it would allow us to move things more incrementally, without taking huge risks.

See Also

Original post:

• https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383

[jmakowski1123]

Explore relation to: https://github.com/openedx/platform-roadmap/issues/2

[ormsbee]

I created a scratch repo to test some of these ideas in (there is some hashing out of the data model for the core publishing app). A number of the structural ideas were adopted from a Django 2021 talk on Scaling Django to 500 Apps.

Some things that came up during the prototyping and discussion:

Phasing

Doing the Publishing portion of this is mandatory, because everything else builds off of it. After that, doing partitioning and composition is probably the best bet, since it's relevant for v2 content libraries and the effort estimation API.

App layout and conventions

• The core set of apps described above go in openedx_learning/apps/core.
• Due to the flat app namespace, the app names will be prefixed, e.g. learning_composition instead of composition.
• Top level core apps have no views or templates.
• Top level core apps have data models that are intended to be extended by other apps
  • For instance, learning_publishing will have a LearningContext model that is general for all learning contexts (courses, libraries, etc.). An app can attach course-specific data to this by making their own Course model that has a foreign key to LearningContext.
  • This might be something we can use Django's contenttypes framework for. See this example.
• Apps are nested. So we might have a learning_composition/learning_composition_xblock
  • Though the fact that we need to put the whole globally-namespaced dir name means we could put it elsewhere and still know that it conceptually belongs as an implementation/extension of learning_composition. For instance, we could do openedx_learning/apps/xblock/learning_composition_xblock to group functionality specific to XBlocks
• This secondary layer is where we have views/REST APIs.
• How do we feel about public APIs for nested apps? Potentially more coupling. Maybe only allow for the second layer?
• Should apps always provide their own views, or should there be a unifying app for versioned APIs–i.e. "rest_v1" as its own app that works by using in-process APIs? Too far against the grain?

Other interesting ideas to experiment with from "Scaling Django to 500 Apps"

• Very granular apps–so moderation tools would be in a completely separate app from the student-facing tools for a given feature. This also makes it easier to separate permissions.
• Activating features via INSTALLED_APPS instead of via feature flags.
• Using Django's autodiscovery tools to find other types of modules that we have conventions around, e.g. data.py, api.py, tasks.py.
• Being extremely strict about what goes inside a given module–e.g. models.py files have only models, and never helper methods or other utilities.

Dependencies

• Avoid circular dependencies
• Top level core apps can depend on each other (everything depends on learning_publishing), but no cycles
• Nested apps (either literally sub-dirs, or conceptually like openedx_learning/apps/xblock/learning_composition_xblock) can depend on the top level core apps.
• Apps don't depend on child apps directly. Apps can implement plugin interfaces that child apps implement–e.g. an app that holds the logic for course outlines can have child apps that implement specific outline processors.
• Don't depend on OpaqueKeys
• Look at import-linter to enforce some of these rules internally.

Data modeling details

• Need to be able to specify a case-sensitive field that's really case sensitive in MySQL, which is non-trivial to do in a way that doesn't break SQLite (and other databases). This is something @pomegranited is looking into for Blockstore.
  • Can't use a BinaryField at the moment because indexing for BinaryFields doesn't work for MySQL in Django.
• We'll probably want to capture whole versions–this will help with previews/masquerading, and be useful for error and content rollback situations. We don't need infinite history, but having multiple versions will be useful.

Lifecycle/Upgrade Considerations

• Apps can start life in edx-platform and migrate into this repo at a later point.
• We might be able to get to a point where releases of this repo are versioned and edx-platform pins to the latest stable.

[ormsbee]

Possible areas of exploration on top of this smaller core:

• Lightweight, standalone system with no server dependencies (i.e. just using SQLite or even raw file system). Target audience being relatively small, in-person courses, rather than MOOCs.
• Supporting other assessment types (H5P, QTI, PrairieLearn, etc.).
• Large-scale content-library marketplace system.