search

openedx / edx-platform

The Open edX LMS & Studio, powering education sites around the world!
https://openedx.org
GNU Affero General Public License v3.0
7.11k stars 3.8k forks source link

[DEPR]: Paver #34467

Open kdmccormick opened 3 months ago

kdmccormick commented 3 months ago

:information_source: This DEPR ticket considers the Paver commands in two groups:

Timeline

Communicated

Operator-facing commands: 2023-03-08 (thread) Internal CI commands: 2024-05-01 (thread)

Acceptance

Operator-facing commands: 2023-05-03 Internal CI commands: 2024-05-21

Replacement available

Operator-facing commands: 2024-05-07, Redwood Internal CI commands: 2024-11-09, Sumac (Target)

Earliest removal

Operator-facing commands: 2024-11-09, Sumac Internal CI commands: as soon as replacement is available

Rationale

edx-platform historically handled its build scripts with paver:

Paver is a Python-based software project scripting tool along the lines of Make or Rake. It is not designed to handle the dependency tracking requirements of, for example, a C program. It is designed to help out with all of your other repetitive tasks (run documentation generators, moving files about, downloading things), all with the convenience of Python’s syntax and massive library of code.

Using paver has a few problems:

For the Paver Asset commands in particular, more depth is provided in the 'Reimplement edx-platform static asset processing' ADR.

Replacement

Module Replacement
assets New static assets guide
quality Directly invoking tools (pylint, etc.) when possible, Makefile targets otherwise
test_js Makefile target
prereqs pip
docs Makefile target
i18n Makefile targets for common tasks; i18n_tool for advanced tasks
servers Tutor (or the deprecated Docker-based Devstack, or bare-metal development

Migration

To ease migration for Paver users, in Redwood, each Paver Asset command will simply proxy to its replacement command, and will raise a deprecation warning explaining the new command that it is running.

By Nov 7th 2024, operators will need to have switched to the new commands, taking the steps indicated below by ACTION REQUIRED.

Requirements (Operators: ACTION REQUIRED)

In Redwood and earlier, Paver and its dependencies were included in requirements/edx/base.txt

Starting in Sumac, these dependencies will be removed from requirements/edx/base.txt. Instead, operators will need to install:

Operator-facing commands with known users (Operators: ACTION REQUIRED)

These replacements are production-ready as of May 7th, for Redwood.

Module Known Users Before After
assets Ansible, Devstack paver update_assets npm run build && ./manage.py lms collecstatic --noinput && ./manage.py cms collectstatic
assets None paver process_xmodule_assets No longer needed - no replacement
assets Tutor, pavelib paver compile_sass npm run compile-sass
assets pavelib ./manage.py [lms/cms] compile_sass npm run compile-sass
assets pavelib paver webpack npm run webpack
assets Devstack paver watch_assets npm run watch
prereqs Ansible, Devstack, pavelib paver install_prereqs pip install -r requirements/edx/base.txt -r requirements/edx/assets.txt && npm clean-install
prereqs pavelib paver install_node_prereqs npm clean-install
prereqs pavelib paver uninstall_python_packages Will not be replaced
prereqs pavelib paver install_coverage_prereqs pip install -r requirements/edx/coverage.txt
prereqs pavelib paver install_python_prereqs pip install -r requirements/edx/base.txt

Operator-facing commands without known users

These commands have been replaced for a long time. We don't know of any users of them, except the old Vagrant Devstack, which was deprecated 8 years ago.

Module Before After
docs paver build_docs make docs
i18n paver i18n_validate_gettext which xgettext
i18n paver i18n_extract make extract_translations
i18n paver i18n_dummy i18n_tool dummy && i18n_tool generate
i18n paver i18n_generate i18n_tool generate
i18n paver i18n_generate_strict i18n_tool generate --strict
i18n paver i18n_clean make clean_translations
servers paver check_settings None
servers paver lms ./manage.py lms runserver
servers paver studio ./manage.py cms runserver
servers paver run_all_servers tutor local ...
servers paver devstack tutor dev ...

Internal CI commands

These commands are only used by the upstream openedx project for CI. They will replaced before Sumac. Operators do not need to take action.

Module Replacement Status Before After
quality This is a no-op paver find_fixme Remove
quality Need to implement paver run_eslint eslint
quality Need to implement paver run_stylelint make stylelint_js
quality Need to implement paver run_xsslint make xsslint
quality Need to implement paver run_pii_check make pii_check
quality Need to implement paver check_keywords make check_keywords
quality Ready to use paver run_quality pylint
quality Ready to use paver run_pylint pylint
quality Need to implement paver run_pep8 pycodestyle
js_test Need to implement paver diff_coverage make diff_coverage_js
js_test Need to implement paver test_js make test_js
js_test Need to implement paver test_js_run make test_js
js_test Need to implement paver test_js_dev make test_js MODE=browser

Django settings for Asset commands (Operators: ACTION REQUIRED)

In order to reimplement the Paver Asset commands without Python/Django, we are changing how several settings are configured.

The following Django settings are becoming read-only mirrors. If you override them now, remove your overrides:

The following Django settings are being removed. If you override them now, remove your overrides:

The following new environment variables are available. Set these in your environment, in place of the overrides you removed above:

Note: If you previously set the STATIC_ROOT_BASE Django setting to /blah, then you should now set the LMS and CMS environment variables as so:

Deprecation

In time for Redwood, deprecation warnings will be added to all edx-platform Paver Asset commands.

Removal

### Tasks/PRs
- [ ] https://github.com/openedx/edx-platform/issues/31624
- [ ] https://github.com/openedx/edx-platform/pull/31810
- [ ] https://github.com/openedx/edx-platform/pull/31934
- [ ] https://github.com/openedx/edx-platform/pull/31790
- [ ] https://github.com/openedx/edx-platform/pull/32717
- [ ] https://github.com/openedx/edx-platform/pull/32767
- [ ] https://github.com/openedx/edx-platform/pull/32804
- [ ] https://github.com/openedx/edx-platform/pull/32823
- [ ] https://github.com/openedx/edx-platform/pull/32866
- [ ] https://github.com/openedx/edx-platform/pull/34318
- [ ] https://github.com/openedx/edx-platform/pull/34468
- [ ] https://github.com/openedx/edx-platform/pull/34554
- [ ] https://github.com/overhangio/tutor/pull/1042
- [ ] https://github.com/openedx/edx-platform/pull/34829
- [ ] https://github.com/openedx/edx-platform/pull/34830
- [ ] https://github.com/openedx/edx-platform/issues/34845
- [ ] https://github.com/openedx/edx-platform/pull/34832
- [ ] TODO update all edx-platform documentation
- [ ] TODO update external documentation
kdmccormick commented 2 months ago

I have updated the Django settings to clarify that we will also be removing the STATIC_ROOT_BASE Django setting.

kdmccormick commented 1 month ago

Now that this is accepted, I'll split the removal into three steps:

Step 1 is free of downstream contingencies. It is safe to happen at any time.

For Step 2, we'll coordinate with early deployers (2U) so that they have time to update their deployment scripts to avoid the assets and prereqs commands. This step will deliver a speed-up to the production Tutor build.

For Step 3, we'll need to move the remaining quality/js checks out of Paver, uninstall Paver, and a do bulk sweep through the docs and update all instructions that still specify "paver".

feanil commented 1 month ago

FYI, I created https://github.com/openedx/edx-platform/issues/34834 to handle a specific github workflow around paver assets that also needs an enhancement.

kdmccormick commented 1 month ago

That workflow update ^ is now handled.

Here is an epic to handle the remaining upstream CI updates:

kdmccormick commented 1 month ago

As of May 7th 2024, the operator-facing Paver commands have been fully replaced with npm & django commands. The replacement commands are being invoked in Redwood (directly) and on edx.org (via legacy Paver wrappers). Given that, we're targeting Nov 7th 2024 for removal of the Paver wrappers, or earlier if our first deployers (2U) indicate that they're ready. I've updated this ticket's description and added "Operators: ACTION REQUIRED" wherever I believe operators need to migrate; I'll make sure those land in the Sumac release notes too. Operators, let me know if there's anything I can do to clarify. Thanks!

timmc-edx commented 4 days ago

The ticket currently says under Replacement Available « Internal CI commands: 2024-11-09, Sumac (Target) » but that's after the targeted removal date of Nov 7. It sounds like maybe the replacements are already available, though?

feanil commented 4 days ago

So Kyle broke this out into a couple of different sections above (different tables). The Commands with known users (Operators: ACTION REQUIRED) and the Commands without known users sections have replacements that have already been implemented as of right now. In some cases the "replacement" is just calling the underlying command directly.

The unfinished section is the Commands used by internal CI where some have been implemented but some still remain. Let me know if that doesn't answer your questions.

kdmccormick commented 3 days ago

What @feanil says is right. I've tried to amend the description to be a bit clearer. In retrospect, it may have been clearer to just have two DEPR tickets, but doing that now would probably just add confusion. @timmc-edx let me know if it's still unclear.

timmc-edx commented 2 days ago

I see now, thanks!

dianakhuang commented 2 days ago

Just putting it here as a note for something that I/someone else could do: would be nice to create a make command for running asset processing, because otherwise I will forget the correct way of doing it.

feanil commented 2 days ago

Just putting it here as a note for something that I/someone else could do: would be nice to create a make command for running asset processing, because otherwise I will forget the correct way of doing it.

FWIW, it's just npm run build or npm run build-dev, that does all the things and aligns with our MFEs which do npm build to build out the serialized assets.

Also there is a full guide in the docs: https://github.com/kdmccormick/edx-platform/blob/master/docs/references/static-assets.rst

dianakhuang commented 2 days ago

Oh, awesome! I have been around too long and of course I didn't think to read the doc. 😅