[DEPR] Devstack

[kdmccormick]

Timeline

• Proposed: 2022-03-01

• Updated Acceptance Date: 2024-03-12

• Archived: 2024-03-19 (pre-Redwood)

  • Anyone wishing to continue to use devstack will have to make their own forks in order to make changes or keep it updated/maintained.
  • 2U will likely maintain its own fork, but there will be a number of 2U-specific changes and will not be open to outside development. It is possible that this fork will be made private.

• Removal of devstack support from application repositories: 2024-10-10 (UPDATE SEPT 2024: This may change. See the latest comments).

Replacement

The replacement for Devstack is Tutor. Tutor is "the Docker-based Open edX distribution designed for peace of mind". It is both a deployment tool and a development environment. Tutor has been the only community-support production deployment method since Maple, although the Open edX docs and website do not officially recommend it as a development environment yet.

Rationale

Devstack has several issues:

• It is not pluggable. This leads to two issues:
  • It is large. Out of the box, it comes with a dozen IDAs and another dozen MFEs; new Open edX developers generally needs two IDAs (LMS and CMS) a few MFEs. In the past couple years, improvements have been made such that only a subset of IDAs are run by default. Still, cloning and provisioning are bound by the full weight of all services.
  • Some IDAs like enterprise-catalog, in order to avoid being part of the large Devstack core, make a plugin-esque docker-compose.yml file that hooks into Devstack. This complicates the developer experience on those IDAs and also makes it harder to modify Devstack itself (because of the implicit plugin interface that such IDAs are using).
• Its images are slow to download. The Docker images used by devstack are built by running Ansible in a Dockerfile instead of using native Dockerfile commands. This means that the images are large and make poor use of the Docker layer cache, and thus take a long time for Devstack users to download.
• Many users find it difficult to diagnose and fix Devstack errors. Often, they find it more time/effort-efficient to destroy their volumes and provision from scratch instead of fixing the immediate issue.
• Provisioning takes a long time. This exacerbates the previous issue.
• There are several "upkeep" steps that must be independently done on a regular basis in order to keep Devstack working reliably. These include git-pulling devstack, git-pulling multiple app repos, pulling Docker images, and running migrations.
• Setting up a service in Devstack is completely tangential to setting it up for production deployment. Once it is set up, it takes attention and effort to keep the setup consistent between Devstack and production.

Tutor shows promise in many of these areas:

• It features a documented plugin API. Many IDAs (forums, discovery, credentials, et al) are implemented as plugins to Tutor. The core of Tutor includes just LMS and CMS, and there is even work being discussed to factor those out into plugins.
• Tutor's Dockerfiles are written using native Dockerfile syntax and are intentionally designed to be as small and cache-friendly as reasonably possible.
• Tutor has actively-maintained documentation including a Troubleshooting section. Beyond that, Tutor has a dedicated forum community where maintainers and users help each other. Tutor also makes a point to have a consistent, documented, and debuggable CLI. For example, the tutor dev segment of the CLI is implemented entirely on top of docker-compose and transparently prints out the underlying docker-compose commands that it runs to aid in user debugging, using terminal colors to call out commands vs tutor logging vs actual output.
• Once Tutor is installed, provisioning involves one command (tutor local quickstart) and runs fairly quickly. Provisioning can be safely re-run without destroying existing data.
• Because Tutor uses the code in the Docker images by default, the user only needs to regularly git-pull the Tutor repo and the repos that they host-mount in, if any.
• Since Tutor is also the community-supported deployment method, one can support development and and production usage of an IDA all in a single plugin. Furthermore, using Tutor for both dev and prod allows operators to keep their dev and prod environments very similar with less effort.

For areas where Tutor could be improved, there is an ongoing Tutor Adoption Initiative which aims to support the transition from Devstack to Tutor through a mix of education, plugin development, and core Tutor improvements. Within the initiative roadmap, the "Dev Quality-of-Life" epic contains stories that should ideally close any existing gaps between Tutor and Devstack.

Migration

Many developers can start using Tutor now, and many already are). Particularly, developers whose projects are encompassed by the LMS, CMS, MFEs, and IDAs covered by existing official plugins can generally transition their workflows over to Tutor. On the other hand, for teams that run IDAs with less community adoption (notably, teams at 2U), several new Tutor plugins will be needed in order for Tutor to fully replace Devstack.

There are no plans to help users migrate data from a Devstack instance to a Tutor instance. We recommend that folks start fresh when switching from Devstack to Tutor.

Deprecation & Removal

Deprecation steps

• Update the development page on open.edx.org to recommend Tutor instead of Devstack.
• Ctrl+F the docs for "Devstack", and where appropriate, update them to point at Tutor.
• Add a deprecation warning to devstack's readme
• Make devstack's dev.up Makefile target print a deprecation warning
• Update the configuration docker role's readme with a deprecation message.
• In the description of https://dockerhub.com/u/edxops, warn that all images in the org are deprecated.

### Removal steps
- [x] Archive https://github.com/openedx/devstack 
- [ ] Update OEP-10, which still mentions "devstack" as a supported distro
- [x] https://github.com/openedx/edx-platform/issues/34430
- [ ] Scour our docs for other references to devstack; update them to Tutor.
- [ ] Delete the various `devstack.py` and `development.py` settings files unless otherwise depended upon.
- [ ] Delete the various `docker-compose.yml` files and Makefile targets that some IDAs (eg enterprise-catalog) use in order to integrate with Devstack.
- [ ] Delete the `docker/` role (incl. Dockerfiles) from github.com/openedx/configuration.
- [ ] Follow up: Decide whether or not to keep the Dockerfiles that exist in various repos, which aren't used in Tutor.

[kdmccormick]

Relates to https://github.com/openedx/build-test-release-wg/issues/138

[dianakhuang]

Since this is primarily a 2U project right now, we are planning on announcing this and gathering feedback.

2U is still planning on using this for the near to medium term until we're confident we can migrate successfully onto Tutor, but we can move it out of the openedx GitHub org to minimize confusion.

[kdmccormick]

In general, sounds great!

Only thing is... MFE development currently sucks in Tutor >.< So nearly all frontend devs still use Devstack when they're trying to change MFEs.

Brian Smith is actively working on rectifying this, with promising results. Would you be willing to delay the deprecation further until Tutor has MFE dev sorted out?

[dianakhuang]

Yup! I think we're still trying to understand the requirements needed here. I'll push out the acceptance date for another few months.

[dianakhuang]

Discuss posting: https://discuss.openedx.org/t/deprecation-devstack/10703

[kdmccormick]

@dianakhuang So, I know I had originally been a proponent of deprecating both configuration and devstack "into the edx org", under the understanding that if only 2U is using some parts of the project, then it's easier for both sides if we transfer those parts over to 2U. Since then, I've talked about this with my team at Axim, and come to understand that the situation is more nuanced than that.

The code in the openedx org is built from a mix of Axim-owned contributions and community-contributor-owned contributions (licensed to Axim under the CLA). We hold the code in this org for the good of all current & former Open edX contributors and users, and as the project's stewards we have a mandate to maintain ownership and ensure availability of all its repositories, even unsupported repositories.

Our procedure of archiving repos by transferring to the openedx-unsupported org lets us mark a repo as "unsupported" while still guaranteeing the repo's availability and keeping its ownership clear. So, if/when devstack and configuration are deprecated, we'll need to move them there.

[dianakhuang]

@kdmccormick that sounds reasonable! 2U/edX.org should be able to then clone the repositories for our own use case after they are put into unsupported, then?

Obviously, this is our first stab and understanding how this process is going to work going forward.

[kdmccormick]

Yes, you could fork from openedx-supported after that full deprecation & archival process.

Even in that case, though, I'd be curious what we could do to help keep 2U involved with community tooling.

Would you be forking devstack & configuration as a temporary stop-gap until you're able to transition off of them, or with the intent of using them indefinitely?

• If you're using them as a stop-gap, then I bet some other community members are also using them as a stop-gap. It could be worth keeping these repos un-archived until the replacement options (Tutor or otherwise) are viable to all or almost all community members.
• If you plan to use them indefinitely, then I bet some some other community members would see value in using them indefinitely, too, which I would see as an argument for keeping the repos un-archived.

Obviously, this is our first stab and understanding how this process is going to work going forward.

Same here :) learning as we go!

[dianakhuang]

We are going to block acceptance because devstack is still used heavily by people who need to do development on MFEs outside of 2U.

@arbrandes @kdmccormick could you add more context on this?

[kdmccormick]

@arbrandes , I know you mentioned that RaccoonGang are heavy users of Devstack. I wonder if they'd be willing to take on maintenance. If not, and if nobody else is willing to, then we should probably let this DEPR move forward and help MFE developers move towards Tutor.

[feanil]

@dianakhuang given recent changes at 2U, is 2U still willing to maintain this repository? Or should we complete the deprecation and remove this from the openedx org?

[dianakhuang]

@feanil 2U is still willing to maintain this repository for now. We would like to keep in in the openedx org for now if that's okay with everyone else.

[feanil]

I think this is fine for now but since we're trying to move away from devstack for the community, at some point it will make senses to move it out. In the meantime, I'd like to mark this ticket and the repo as deprecated and make sure the relevant messaging exists in the README for this. What do you think?

[dianakhuang]

Yup, let's add some text to the README and move this ticket to the deprecated column.

[feanil]

Cool, is that something you can take on as maintainers?

[dianakhuang]

Yup, I'll try to get on it today and spread the info around 2U so no one freaks out.

[feanil]

We should move this back to Proposed and re-communicate that 2U wants to stop maintaining this as a part of Open edX. Either someone else needs to maintain this for the community or we can archive and 2U and any other interested parties can fork it.

[kdmccormick]

Re-communication happened here: https://discuss.openedx.org/t/deprecation-removal-devstack-update/12384

Aaaand it's archived: https://github.com/openedx-unsupported/devstack

I moved this ticket to public-engineering, since we have a few more clean-up steps before we can close it out.

[kdmccormick]

With the devstack repository archived, the obvious next question is: When will devstack.py, env.devstack, and any other devstack-specific files be removed from upstream repositories? I realize that removing those files will break any forks of devstack, unless the forks adapt by loading the files from somewhere new.

I propose that the files should remain in-place until the Sumac branch cut on October 9th, after which they are fair game for removal. Happy to discuss different dates, a phased removal, or something else. Let me know what you think @dianakhuang @regisb @cmltaWt0 @felipemontoya @robrap @feanil .

[kdmccormick]

^ @brian-smith-tcril @arbrandes @adamstankiewicz , pinging you guys on behalf of the frontend :)

[feanil]

October 9th seems like plenty far int the future, but I think we should put deprecation warnings in those files now with the agreed upon dates so people who touch them have some chance of knowing that they're gonna disappear.

[robrap]

Thanks @kdmccormick. I'm hopeful we can work that into our plan at this point, but tagging @jristau1984 and @spencertiberi, who need to be involved in any deadline agreements. Thanks.

[kdmccormick]

I'd like to settle on a timeline for removal of these auxillary files. If there any objections or alternative suggestions to Oct 9 for the "OK to start removing date", please raise them by the end of next week (Apr 5).

[jristau1984]

At this time, I have no concerns with having the removal work begin after the Sumac cut, currently scheduled on Oct 9.

Can this date be added to the timeline section of the issue above? Thanks!

[robrap]

@jristau1984: Just to ensure you are aware, this is not just busy work, but requires designing a plan for how we can accomplish this.

[kdmccormick]

Can this date be added to the timeline section of the issue above? Thanks!

Done.

When I get a chance, I'll start a draft PR of these changes for edx-platform, so that devstack users can get a sense of which files will be going away.

[kdmccormick]

Deprecation warnings for devstack settings files in edx-platform: https://github.com/openedx/edx-platform/pull/34795

[iamsobanjaved]

Hi @kdmccormick

I’m working on the 2U side to move Devstack-related settings out of the Open edX repositories and into a different location within the edX GitHub organization. While doing this I was wondering how Tutor has done the development environment setup, I noticed that including the LMS, CMS, and other plugins like discovery and credentials, relies heavily on the devstack.py or development.py modules. These services use devstack.py as a base and then modify a few settings variables.

Is there currently any effort underway for the Tutor to stop depending on these files? Or, as mentioned in the Removal Steps, should we refrain from deleting them if they are still in use?

Delete the various devstack.py and development.py settings files unless otherwise depended upon.

Tutor using devstack files as a base:

• https://github.com/overhangio/tutor/blob/master/tutor/templates/apps/openedx/settings/lms/development.py#L3
• https://github.com/overhangio/tutor/blob/master/tutor/templates/apps/openedx/settings/cms/development.py#L3
• https://github.com/overhangio/tutor-discovery/blob/master/tutordiscovery/templates/discovery/apps/settings/tutor/development.py#L1
• https://github.com/overhangio/tutor-credentials/blob/master/tutorcredentials/templates/credentials/apps/credentials/settings/development.py#L1

[regisb]

I hadn't thought about the fact that getting rid of devstack meant removing devstack.py. Do we all agree that we still need platform-agnostic development settings, though? I would expect that devstack.py would be replaced by development.py settings files that provide good defaults for development. If we do, then we can configure tutor to point to these new development modules instead of devstack.py.

[kdmccormick]

@iamsobanjaved thank you for pointing this out, I had not realized that Tutor was based on those files as well.

@regisb I agree, we do need platform-agnostic development base settings.

I can work to develop a platform-agnostic development.py settings file for both CMS and LMS, which both Tutor and Devstack can use. Once that is merged, I can wait 30 days for Devstack and Tutor to switch over to development.py before we remove devstack.py. Does that sound reasonable?

Also, @regisb , Axim has bought local.openedx.io and we can point it (and its subdomains) at 127.0.0.1. Do you have any objection to me using local.openedx.io as the base URL in the development settings file? Tutor could either use it as well, or it could continue to override the base URL to local.edly.io.

[kdmccormick]

Clarification: development.py would be a suitable base for Devstack settings, but it would not work out-of-the-box for Devstack because it would not include the specific ports and hostnames that Devstack needs, and it may not have the exact same toggles that Devstack users expect. So, Devstack would still need a custom settings file for LMS and CMS. In the meantime, I recommend making sure that Devstack has the capability to define a custom DJANGO_SETTINGS_MODULE.

[iamsobanjaved]

Yeah that's what exactly I wanted, a suitable base for dev settings that can be further extended. Also, we are planning to use YAML to override devstack-related settings and It would be nice to have a development.py that can be extended/overridden by a YAML file.

[kdmccormick]

Great. In order to keep development.py straightforward, I am inclined not to load YAML into it, but I can provide a YAML snippet that you could add to Devstack's custom settings module in order to retain YAML support.

[regisb]

Thank you! I'm looking forward to a future with streamlined development settings.

On loading YAML files: I'm also not too inclined on doing that... but I'm sure you'll be able to do that within devstack.py.

It's awesome that we can now switch to local.openedx.io! As we learned previously, the switch is actually very easy. I'll add that to our roadmap (here).