feat: docker-based local work without lms

[marlonkeating]

Description:

Adds a make test-shell which launches a dockerized version of the local dev environment - rather than using a local virtualenv.

~ Working from https://github.com/openedx/edx-enterprise/pull/1467 ~

There are sometimes benefits and conveniences to having a containerized local environment versus setting up and maintaining python, virtualenv, and system dependencies (mysql-dev, etc) natively. Sometimes installing your code into the LMS container extends your testing cycle past your patience. Here is some docker/makefile config to enable make test-shell to launch a containerized environment containing your local development code so you can make test / make upgrade / 'profit' / etc inside a container.

Installation instructions:

If running into issues with make test-shell, try running: docker-compose build --no-cache

Testing instructions:

1. run make test-shell
2. run 'pytest'
3. run 'make validate`

Reviewers:

• [ ] @johnnagro

Merge checklist:

• [ ] All reviewers approved
• [ ] CI build is green
• [ ] Version bumped
• [ ] Changelog record added
• [ ] Documentation updated (not only docstrings)
• [ ] Commits are squashed

Post merge:

• [ ] Create a tag
• [ ] Check new version is pushed to PyPi after tag-triggered build is finished.
• [ ] Delete working branch (if not needed anymore)

Author concerns:

List any concerns about this PR - inelegant solutions, hacks, quick-and-dirty implementations, concerns about migrations, etc.

[timmc-edx]

Can you say more about the problems you're encountering that this helps with? Getting the right Python installed, setting up a virtualenv, etc. or maybe some particular workflow?

To digress a bit, I've been thinking about our cookiecutter situation where we end up adding more and more build/dev code to each repo, but unevenly, and then can't readily upgrade/maintain all the copies. I'm at the point where I'm really reluctant to add more files to each repo if there's any possible alternative—especially when they're likely to diverge over time. An alternative approach would be using git subtrees to pull in the core of the Makefile, dramatically reducing boilerplate.

Bringing this back around, one thing I've toyed with is having Makefile script blocks automatically run in a docker container. I'm curious if having one standard Docker image for all libraries, containing a standardized execution environment, would meet your needs—or if you're looking for something specific to this library.

[marlonkeating]

The goal of this solution was to solve the general problem of manual virtualenv setup and switching (spurred by my specific problem of not being able to get the repo's virtualenv setup instructions to work on my remote devstack).

Using this dockerization solution has served us well in other repositories (https://github.com/openedx/edx-enterprise/pull/1467) (https://github.com/edx/edx-arch-experiments/pull/420), but I heartily agree it would be ideal to use a single docker/makefile template from a centralized location rather that copy-pasting solutions from repo to repo. @timmc-edx's makefile and docker solutions look promising in principle, but I haven't gotten them to work on my environment thus far.

[timmc-edx]

Ohhhhh, yeah. I'm definitely not a fan of all the READMEs that try to tell you how to set up a virtualenv -- I've never followed those instructions ever since figuring out my workflow.

Personally, what works for me is to run python3.8 -m venv .venv-3.8 to create the virtualenv inside the repo (with .venv-3.8 in my global gitignore), and source .venv-3.8/bin/activate when I want to use it. Except I actually have defined an alias so I can write v 3.8 and have it all just work. A couple of repos have problems with an in-repo virtualenv, but the great majority do not.

But I'm pretty sure most of my teammates use the workon command from virtualenvwrapper, which places the virtualenvs in an entirely different place in their home directory! And probably other devs have their own practices.

I really don't like the fragmentation, and putting the "how to do basic Python setup" documentation in the README just makes it hard to maintain. I think we should remove those instructions and either link to a centralized, standard workflow or just make it so devs don't have to manage the virtualenvs directly.

[timmc-edx]

I've filed an issue for changing the cookiecutter-generated README to not have detailed descriptions of how to set up for developing on repos and to instead link to a common page that we can keep up to date: https://github.com/openedx/edx-cookiecutters/issues/396 -- does that cover some of what you're looking for?

[timmc-edx]

@marlonkeating I'm going to close this for now since we haven't heard back in a few weeks, but please let me know if you have further thoughts on what we can do to streamline development.

I also got some feedback from some developers that having repos manage their own virtualenvs would likely cause some difficulty with Tutor, so that would require more thought. I'm definitely going to look into git subtree based approaches to having common tooling across repos, which could include a generic Docker image.