ochan1
commented
12 months ago Thanks, Oliver
Some history and how we use Conda: We use Conda since we couldn't install Python 3.9 on the OCF server and Conda allowed us to have as much of a separate environment as possible without a VM (which was not reliable a few years later).
Now that OCF has Python 3.9, the question is if we still want Conda. In the PR description and code skimming, sounds like Conda should no longer be used. We can consider a few years ago where like people can say "yeah, we don't need to easily upgrade, it won't matter much down the line" until it's too late. Now that we want to update to Python 3.9, Conda allowed that flexibility and option without going through OCF. Maybe in 5 years from now we need to upgrade to Python 3.11 and we wait and wait for OCF, but with Conda we can upgrade ourselves right away.
In summary for Conda:
- As much of a separate standardized dev environment with things as much auto installed as much as possible
- Less reliance on OCF to get Python upgraded or other binaries
At least you were able to try on OCF which is a good thing too
I'll give this a try later if this is the decision to use this among the officers
oliver-ni
Modernizing tooling for future development
relevant slack thread
Replacing conda + requirements.txt with Poetry
Currently, conda is used to manage requirements, and it installs requirements listed in a requirements.txt file. Because conda is mostly an environment manager and not a dependency manager, it doesn't work well for this use case — for one, it's very susceptible to breakage because dependencies aren't locked.
Furthermore, I was unable to get the project even running on my machine without much effort wrangling the environment.
Poetry is the modern standard for Python dependency management these days. It handles locking dependencies to ensure reproducible builds, makes adding and updating dependencies easy, and actually does dependency resolution properly to ensure we have compatible packages, so I've switched us over to that.
It's pretty simple to use — just
poetry installto, unsurprisingly, install dependencies, and thenpoetry shelldrops you into the project's virtualenv. It doesn't conflict with your Python installation and/or outside-of-Python stuff (as conda does due to its nature as an environment manager).Disclaimer that you can use Poetry with conda if you want. Really, they're designed to do different things. You can use conda to manage your Python versions / global environments and Poetry to manage project dependencies.
Bringing dependencies up-to-date
Since I changed the system used for dependency management, I decided to go through and update all our dependencies too (well, some of them weren't pinned properly in the first place, so I didn't even know which versions those were running).
Django is now at version 4.2 LTS. I also updated bleach, gunicorn, django-autocomplete-light, and djangorestframework. I removed the dependency on whitenoise and pillow which were not being used. Additionally, I noticed that someone seemed to have started implementing social auth, but never finished it, so I stripped that code and removed the dependency on social-auth-app-django. If we want to do that, we can start again in the future.
I've gone through all the release notes and updated our code where libraries had breaking changes. With some preliminary testing, I'm pretty sure everything works, but it would be nice if someone else could test a bit too.
This is not only a nice touch to keep the site secure and updated but also a lot of the versions we were on were plainly just not compatible with my system no matter what I tried (especially since I am on Apple Silicon). I want to get the repository to a clean, workable state so it's easy for us to work on the site.
Updating the deployment/ci pipeline
Replacing conda affects deployment as well. I've installed Python 3.9 and Poetry outside of conda on OCF apphost, and updated all the deployment, run scripts, and CI workflow to use Poetry instead.
The commands are still the same:
fab deploywith theHKNWEB_MODEenvironment variable set to eitherdevorprodto deploy to a local server or the production OCF server respectively.Additionally, I reworked the GitHub Actions workflow to automatically deploy after running tests. This should make it a lot easier to update the website as every push to
masterwill automatically deploy without needing to manually run the script. You can also see the status of the deployment on GitHub due to the new-ish environments feature.Adding an optional Nix flake
Nix is a cool build system that puts an incredible emphasis on declarative, reproducible builds as well as declarative configuration. I've added an optional flake in the root of the repository. The flake reads the Poetry configuration and builds a development shell with all the dependencies (including native dependencies!) needed to run the project automatically.
I'm not expecting everyone to adopt Nix, which is why it's not required to run the project. The way I set it up, the Nix config is derived from the Poetry config, so you can just use Poetry directly. It's just a convenience for those who use Nix.
But Nix is cool! You should check it out :) :)
For reviewers
Hi! If you're able to take the time to review, let's try setting up a dev environment with the new tooling :)
modernizationbranchpoetry install[1]poetry shellto drop into the project's virtualenvpython manage.py migratepython manage.py runserverand you should have a working server at http://localhost:8000!If this doesn't work for you, please let me know and let's iron out the issues :)
I still need to update the Wiki but I don't think that's tracked in Git, so I'll update it when the PR is merged.
[1] This should usually work. However, if Poetry can't find the right version of Python, you may need to push poetry with
poetry env use /path/to/python3.9.