search

openedx / public-engineering

General public issue repository for the Open edX engineering community
4 stars 2 forks source link

[DEPR]: Remove Learner View in Insights, Data API, and Pipeline #36

Closed ashultz0 closed 9 months ago

ashultz0 commented 2 years ago

Editorial note: This deprecated a particular view in Insights ("Learner View"), plus that view's supporting code in other analytics repos. It didn't entirely deprecate Insights and all associated analytics repos. -Kyle

Proposal Date

3 March 2022

Ticket Acceptance Date

31 March 2022

Technology Removal Date

May 1 2022

First Open edX Named Release Without This Functionality

Olive

Rationale

The learner view in insights and associated data API and pipeline are expensive to operate and provide data out of alignment with the rest of analytics. It is so detailed that it provides too much individual data to be viewable for a course with more than a few students, and that data creates PII risk.

Most of analytics is aggregated data. The learner view contains individual information including usernames and emails. This means that the learner view includes sensitive PII. This creates unnecessary security concerns Insights as well as the entire pipeline and API.

To support this data requires an expensive pipeline job and an elasticsearch cluster ( see https://github.com/openedx/public-engineering/issues/16 ). On edx.org, data supporting this view is roughly as as the rest of Insights combined.

Removal

Insights code supporting the learner view such as: https://github.com/openedx/edx-analytics-dashboard/tree/master/analytics_dashboard/learner_analytics_api

Learner view data API code such as https://github.com/openedx/edx-analytics-data-api/blob/master/analytics_data_api/v0/views/learners.py as well as elasticsearch support code and the corresponding calls in https://github.com/openedx/edx-analytics-data-api-client

Learner view pipeline support such as https://github.com/edx/edx-analytics-pipeline/blob/master/edx/analytics/tasks/insights/module_engagement.py and elasticsearch support.

Replacement

There is no replacement currently planned.

Deprecation

No response

Migration

No response

Additional Info

No response

kdmccormick commented 2 years ago

Forum post: https://discuss.openedx.org/t/deprecation-removal-learner-view-in-insights-data-api-and-analytics-pipeline/6788

dianakhuang commented 2 years ago

This is still under discussion after the accepted date. See the Discourse post for more details.

jmbowman commented 2 years ago

The Discourse post resolved at it being ok for us to remove this, just need to actually rip it out now.

pdpinch commented 1 year ago

When I was looking through the commit history to prepare the olive release notes, I found the following PRs, which suggest that this feature has been removed.

Can this issue be closed, or at least updated?

ashultz0 commented 1 year ago

It has been removed from the frontend parts of the system but there are some reports in enterprise code tied to the back end structures that have to be rooted out as well to close this and then the back end itself can be removed.

As far as Olive is concerned this feature is not available, but the code is still being mined out.

bmtcril commented 1 year ago

Is this still being worked on, or can this ticket be closed out?

ashultz0 commented 1 year ago

still in the yak shaving stage of trying to get rid of the code that lets us get rid of the code that lets us get rid of the code :(

jmbowman commented 1 year ago

Can we ticket up and link that chain of code removal, or at least the task that's currently blocking this?

ashultz0 commented 1 year ago

I'm a bit detached from where this is now because I've changed groups so I summon @alangsto who is another person who knows something :)

alangsto commented 1 year ago

Cosmonauts has tickets to remove the learner view, but the work has not been prioritized. If this work should be prioritized, I can bring it up during our next grooming meeting.

kdmccormick commented 1 year ago

Could you link the tickets here Ali? I understand that they're probably 2U-internal, but I'd like to tie all threads together here, whether private or public.

alangsto commented 1 year ago

Here is the link to the epic: https://2u-internal.atlassian.net/browse/MST-1428

dianakhuang commented 9 months ago

This is being closed in favor https://github.com/openedx/public-engineering/issues/221 , since that other ticket will cover these views.