search

openedx / platform-roadmap

Tracking the maintenance, enhancement, and advancement of the Open edX project.
11 stars 0 forks source link

Product Discovery - Aspects. MVP #255

Open jmakowski1123 opened 1 year ago

jmakowski1123 commented 1 year ago

Panels for quick reads of real-time learner and course data:

github-actions[bot] commented 1 year ago

Thanks for your submission, @openedx/open-edx-project-managers will review shortly.

jmakowski1123 commented 1 year ago

Next steps: Jenna and Brian can use this ticket to refine OARS MVP

jmakowski1123 commented 1 year ago

Collaborators: -Felipe to help bring the big needs from Spanish proposal into fine-grained use cases

e0d commented 1 year ago

@felipemontoya I understand from the DWG meeting that you are planning to adopt OARs as the data platform for the Spanish Consortium project. So, their campus related analytics would be based on that foundation. Is that accurate?

felipemontoya commented 1 year ago

Yes that is accurate. We started to put engineering hours into making OARS land a V1 that is suitable for prod environments.

I need to transfer the use cases from the Spanish tender here and Brian told me it would be good to have this by the end of the month because it will be compiled with use cases from other sources.

felipemontoya commented 1 year ago

Abstract

The Spanish OpenedX for Campus Project aims to improve the analytics capabilities of the Open edX platform by supporting the OARS project and creating custom metrics and dashboards for specific use cases. The project requests aggregated data dashboards that show metrics grouped by category and filtered by courses and time, individual access dashboards that allow students to compare themselves with other students in the course, and other custom metrics that can be created from xblocks or plugins. The project also plans to write optional extensions that can plug into OARS without forking the underlying plugins.

Context & Background

The Open edX platform has used Insights in the past to calculate analytics for educators who want to examine one course at a time. However, Insights has some limitations when it comes to analyzing multiple courses together, more granular data, or individual students compared with the course average.

Insights also relies on batch data processing, which prevents near-real-time analytics from reaching the educators for timely intervention.

Scope & Approach

The Spanish consortium tender requested a mechanism to publish real-time log events in a data lake for big-data processing. This coincided with the launch of the Open edX Analytics Reference System (OARS) project. As the developers selected by the consortium, we decided to support the OARS project and try to fit our requirements into its capabilities or extensibility options.

However, we understand that some of the dashboard, graphs and metrics that we outline in the next section may be too specific for OARS. Therefore, we plan to write them as optional extensions that can plug into OARS without forking the underlying repositories. It is possible that more extension points will be required to achieve some of this goals.

Requested Dashboard and Metrics

  1. Aggregated data dashboards:

A series of dashboards with metrics that can be filtered to one or more courses, with start and end times. Filtering this should make all the stats below be updated.

The metrics that follow would be grouped by category. Here are some examples of metrics. The complete list is published in the doc: https://docs.google.com/document/d/1NPYkn7QCJ1f6JUSz0Wr3uCjswLYFOI1o6BKXjAjk3Aw/edit#

  1. Individual access dashboards:

This series of dashboards have in essence the same type of data from the aggregated panels, but in this case the idea is that student users are able to compare themselves with the other students enrolled in the course.

For the comparison, we can use different statistical performance metrics such as quartiles. We can also calculate a “score” based on participation and use it as a comparison metric.

We can embed these dashboards in the LMS for students to see their own stats and let instructors see the students individually as well.

  1. Other custom metrics:

The requirement includes the capacity to create metrics for things that exist only in the form of xblocks or plugins. We believe that this would be too specific for this description, but we include the capacity to publish events from a plugin/xblock that can be then further processed by logic also residing in the same or a different plugin.

An example of this is the metric of downloaded course material, which requires the presence of a Course Material tab in the course most likely to be implemented as an xblock.

Milestones and/or Epics

Milestone 1: reaching a usable OARS V1. Milestone 2: extensibility of OARS through plugins. Milestone 3: Definition of common metrics and dashboards. Milestone 4: Creation of Spanish Consortium specific dashboards/metrics in open sourced plugins.

Named Release

Although the first milestone of reaching OARS V1 is likely to be reached short after the publication of Palm. We expect to have something more robust by Quince.

Proposed By

Spanish OpenedX for Campus Project

cc: @bmtcril @ian2012

jmakowski1123 commented 1 year ago

@felipemontoya There are some metrics from the Spanish use cases that require some additional context. I've transferred into an excel sheet and added some extra columns. Column K indicates where we need some additional clarification. Column L indicates some scope restrictions from Brian. Thought we could start by sharing notes/comments in here asynch, and we can always schedule a follow up call if we need to chat through some of these use cases in more detail! cc @bmtcril https://docs.google.com/spreadsheets/d/1_lqmXI33Nlb6C37craFrjpD64Od90IX2ev5vI8NAJiM/edit#gid=1943589061

felipemontoya commented 1 year ago

I commented on the ones I had clarity. I'll ask Nacho more info on the ones missing.

jmakowski1123 commented 1 year ago

Status update 7/20: Market research complete and use cases have been documented, organized and prioritized: https://docs.google.com/spreadsheets/d/1_lqmXI33Nlb6C37craFrjpD64Od90IX2ev5vI8NAJiM/edit#gid=0

Next step: Create an MVP product spec that includes:

-Admin dashboard experience that includes basic reports on instance/org health, growth, stats, and what those reports include -Course team dashboard experience that includes reports on course data, learner progress, and to identify at-risk learners, and what those reports include -Course team experience to pull reports at the problem/assessment level, and what those reports include

Timeline: Drafted by July 28 and presented to the first August Data WG

jmakowski1123 commented 1 year ago

Update 8/15 - Completed a draft MVP spec for reports to be included in the MVP for admins and educators: https://openedx.atlassian.net/wiki/spaces/OEPM/pages/3834281985/DRAFT+Aspects+V1+Product+Requirements

Next step: Define MVP requirements for report visualization (MVP will probably be limited to leveraging visualizations currently available via Superset) and report dashboard access.

sarina commented 2 months ago

@jmakowski1123 I found myself in this ticket for some reason, and I notice the ticket is still open although it was moved to Done on the board. I think it would be clearer if tickets marked as Done in the roadmap were closed. If you agree I can set up that automation, but we'll need to manually close tickets that are already done.