search

openedx / public-engineering

General public issue repository for the Open edX engineering community
4 stars 2 forks source link

[DEPR]: All Insights related code, configuration, docs and repositories #221

Open kdmccormick opened 9 months ago

kdmccormick commented 9 months ago

(@bmtcril wrote this up originally)

This ticket has been accepted, and the various steps will be carried out in the following sub-tickets:

Sub-tickets

Proposal Date

2023-12-13

Ticket Acceptance Date

2024-01-24

Technology Removal Date

Ongoing

First Open edX Named Release Without This Functionality

This will not be tagged for Redwood, even if no other changes happen.

Rationale

Insights as a product has been largely unsupported for several years, and the lack of community uptake on the project offers little incentive for anyone to take over maintenance. Effectively, the software required to run the product is entirely out of support (the project runs on Python 2.7, requirements haven't been updated in 2-3 years, likely because most packages no longer support 2.7).

Even without a fully released replacement, we should not be suggesting this as a solution.

Note: This is different from the Insights repository that was deprecated in 2019: https://github.com/edx-unsupported/insights

Removal

Insights is a large project that touches many things. This list is known to be incomplete and will grow over the deprecation project, but should give a good cross-section of the work:

The following repositories should be moved to the edx-unsupported organization and clearly labeled as deprecated:

References to those projects and Insights itself exist in at least the following locations:

edx-platform has settings-gated links to Insights, especially in the Instructor Dashboard, we should make sure that all of those settings make sense for plugging in replacements systems. Settings in edx-platform are:

Additionally, calls to the Analytics API should be removed or abstracted such that they can feasibly be reused by replacement systems.

The following repositories (and possibly more) have references to Insights that need to be understood, and made generic enough for replacements to use or (eventually) removed:

Localized strings exist in at least edx-platform and ecommerce that need to be revisited or removed.

Replacement

There are community based replacement systems such as Cairn, and home grown solutions that providers have been using instead of Insights for years. While the new Aspects system is not yet generally available, it should be by the time this ticket is ready for work.

Even if there were no viable replacements, it is neither reasonable nor safe to suggest that Insights, in its current state, is a viable solution for operators to adopt.

Deprecation

Given the scope of the work of this project and potential difficulties of migration for large operators that rely on Insights, a tiered deprecation is suggested, in this order:

This is where breaking changes start...

Migration

For almost all installations no migration will be necessary. For those using Insights the migration will depend on whether they wish to continue using it or move to a new system. The steps for migrating to different new systems are outside the scope of this DEPR. The rest of these steps outline the process of keeping Insights running while this ticket is being worked on, and beyond.

During the non-breaking-changes steps the migrations would be things like updating settings / setting names. Deployment steps for the project may need to change if the deployment method doesn't follow GitHub redirects. Documentation links may need to be updated to point to provider-specific locations.

For breaking changes, most things in tubular, devstack, and configuration are either not working/unused or able to be pulled out of those locations and run from different locations.

Removal of code paths which cannot be generalized (such as explicit calls to analytics-api) can be delayed until suitable pluggability functionality can be added, or the code can be forked to support the use case if the Insights functionality is critical to the repository.

Additional Info

See https://github.com/openedx/public-engineering/issues/36 for additional context

bmtcril commented 8 months ago

This has had no dissenting voices in the comment period and has been moved to accepted. Next step is to write up a more detailed plan!

kdmccormick commented 8 months ago

@bmtcril It's likely that these repositories:

will be deprecated out of the openedx org in the near-to-medium future, and then continued to be used by 2U on a fork. So, we could probably leave the Insights-related stuff in those repositories for now, which would save us some time, and save 2U time since they wouldn't need to figure out how to plug the Insights stuff back into those.

pomegranited commented 7 months ago

@kdmccormick @bmtcril

  • tubular

The community still needs this one.. it runs the user retirement pipeline, cf user_retirement_pipeline/README, tutor-contrib-retirement.

And I'm happy to help out with this deprecation where needed, please let me know :)

bmtcril commented 7 months ago

@pomegranited @feanil is heading that up, but the scripts are moving here: https://github.com/openedx/edx-platform/pull/34063

feanil commented 7 months ago

FYI, I posted https://github.com/hastexo/tutor-contrib-retirement/issues/32 and we're not currently planning to update configuration as a part of the retirement code move so if that's valuable someone else should take that on.

robrap commented 7 months ago

This is a private link to a 2U ticket to look into this DEPR. I'm just noting it in case there are any updates that others in 2U want to communicate back to the team that will eventually be looking into this.

MichaelRoytman commented 7 months ago

Hello from 2U 👋!

As Robert mentioned, we'll be looking into how 2U adapts to these changes shortly. I'll be the point person for the discovery. Because the linked Jira ticket is private, please feel free to tag me on any relevant tickets or if you need my attention on anything. I'll be following this ticket for any updates. I'll be reviewing this DEPR ticket over the next few days, and, over the longer term, working to formulate a plan for the future of an insights product at 2U.

I'm anticipating having a lot of questions about this project. Is this the right place to leave them?

Thank you!

bmtcril commented 7 months ago

Hey Michael! The initial work is in the sub-ticket: https://github.com/openedx/public-engineering/issues/227 so you may wish to follow that one as well. Here is a fine place to discuss, or #wg-depr-slash-n-burn in Slack. I'm happy to do a synchronous meeting as well if that helps anything.

HassanJaveed84 commented 6 months ago

Note that these repositories are still needed to support partner contractual obligations. We have running pipelines which make use of these and are unrelated to insights.

bmtcril commented 6 months ago

Hey @HassanJaveed84 ! I think your best bet is to fork them back into the edx org and run from there. If we move them to the openedx-unsupported org, as is currently the plan, it's likely that the pipelines will continue to work if they follow the Github redirects or are pulling from PyPI, but obviously changes can't be made there.

kdmccormick commented 4 months ago

@bmtcril Is this good to close out, or was there post-archival cleanup that you wanted to track with this DEPR ticket?

bmtcril commented 4 months ago

From the list above, these things still need to happen:

There is a pre-task of trying to identify all of those places, but I'd rather not close this one out until those are done.

bmtcril commented 2 months ago

Note: There are a number of places with historical or references to old code that can stay, or are used in test data. All references in these places seem fine to remain: