Modernize Python Development

[ryneeverett]

Our contributing guide suggests the sole use of virtualenv which from my perspective is a bit dated. We should add support and documentation for newer approaches which provide more reproducibility.

I personally use poetry and nix-shell and I know at least one person uses pipenv. There may be additional tools out there that I haven't heard of. While I was hoping to ignore this issue until the python community developed a consensus on development tooling there doesn't seem to be any hope of that happening soon. In the near term I would suggest we be open to merging support (and documentation!) for any/all development tools.

I believe it should be possible to maintain a single pipfile that can be used by both poetry and pipenv. Regardless, I'd like to see a single update mechanism and we should also consider the process by which we'd like to keep the versions of lockfiles up to date. (Perhaps a quarterly CI bot?)

I'd suggest the following checklist for reviewing new dev env PRs:

• documentation
• CI testing
• plan to keep files up to date

Edit: It appears from the gitignore additions that @d-miketa is using venv. This might be worth considering as well.

[djmitche]

I saw the tox.ini in the repo and ran tox to run the tests. First, it only uses old versions of Py3 which I don't have, so I hacked in a py39-jira200. Having added that, loading the tests fails with #829, meaning no tests run.

Looking at the github actions, it seems like tox isn't used in CI, and CI doesn't even run python setup.py test like tox does. So maybe a place to start is bringing those two things in line?

Also, I noticed that CI is only testing on aarch64 and ppc4le. For a mostly-Python tool, that seems a bit esoteric, and especially so since it does not include the common x86 architecture. Maybe CI would run more quickly if the full matrix ran on x86 and only the latest python version / jira version ran on the other architectures?

[djmitche]

Confirmed: tox doesn't run the tests correctly :/

[ryneeverett]

I opened a new issue to address your concerns because I'd like to limit the scope of this one to the development environment. While you inspired me to finally write these thoughts down I've known we need to address the reproducibility problems with our development environment approach for some time even though that wasn't your issue.

[djmitche]

Got it, thanks -- sorry for misunderstanding this bug.

[ryneeverett]

From my perspective there are some blockers to tracking a pyproject.toml file, which seems like the next step. I'm currently using poetry and would like to avoid duplicating the dependency specification more than twice. One of the following issues would need to be resolved to avoid specifying them three times -- once in setup.py, once in pyproject.toml for pip/setuptools, and again in pyproject.toml for poetry:

• Setuptools currently supports toml on an experimental basis, but newer versions of setuptools use the toml file automatically. As far as I can tell major projects are still duplicating all the information between setup.py and pyproject.toml.
• Poetry doesn't support the PEP 621 dependency specification. Therefore we'd have to specify all the same stuff in both [package] and [tool.poetry].

If either one of these was resolved I'd be looking to move forward with pyproject.toml because I'm already maintaining a duplicate specification in my untracked pyproject.toml at the moment.