flake / poetry / pip clarification and cleanups?

[pazz]

This is mainly for @lucc, who worked on the CI setup and the different setup tools:

Running up to the next release, could you clarify which config files we want / need? Currently there are several legacy things from setuptools and others.

• setup.py and setup.cfg ( I suspect we only need one of these two)
• pyproject.toml
• flake.nix and flake.lock
• poetry.lock (is there a separate config?)

could you comment on the above and also where we'd need to keep track of dependencies?

aside:

I have finally managed to install the development version of alot on debian testing and it required a virtual env set up with --system-site-packages. Only this way could I access the system-wide package "python3-gpg" with the official current gpg module.

The installation via poetry as documented now does not work on debian:

• Debian does not come with poetry built-in so docs would have to be adjusted.
• more crucially, I was not able, after installing poetry via pip in my venv, to use poetry install as documented because it falls back to the gpg module from pypi, which is a) outdated and b) does not compile on debian due to hard-coded strings from the stone age.

If I understand well, poetry does rely on pip itself. I would prefer, if that is reasonable, mention pip first, and poetry/flake/* as alternatives. Does that make sense?

Once we converged on this, let's update the docs, and our release-workflow accordingly and tag a release?

[lucc]

I think these are the tools that use these file directly. (nix for example uses poetry in our setup but can also use setuptools)

	setup.py	setup.cfg	pyproject.toml	poetry.lock	flake.{nix,lock}
setuptools	x	x	x
poetry			x	x
nix					x

I would suggest we migrate setup.* to pyproject.toml as this is the new standard for setuptools and the bigger python ecosystem (see PEP 621 and related).

I would also suggest to use either setuptools or poetry as our official python build system, but not both. Now the problem with either one is our gpg dependency. Either we are using the old version from pypi or we need some other environment (distro + --system-site-packages or nix) to sneak a new version of gpg into our python build environment. And that is the main question for me: What is our "default" way of installing alot? The one we advertise in the docs as the first way that users should try. I would still be happy if nix stays as an alternative although that is what I am using exclusively.

The problem with the install instruction àla

1. install python3-gpg with your system package manager
2. create a virtual environment with system-site-packages=true
3. run $TOOL install alot inside the venv

is that we have some indeterminism at step one and this might be troublesome to debug when we receive bug reports.

PS: One alternative that we did not discuss is that we could offer a container (docker, appimage) as default installation/distribution method. Then only the advanced installation instructions would be concerned with the hairy details of the gpg dependency.

[pazz]

First, thanks a lot for this detailed breakdown and constructive comment!

I would suggest we migrate setup.* to pyproject.toml as this is the new standard for setuptools and the bigger python ecosystem (see PEP 621 and related).

great, let's do that then. I am happy to keep the nix files under extra/

I would also suggest to use either setuptools or poetry as our official python build system, but not both.

I would prefer setuptools over poetry based on my instinct to go with what is more established. One benefit is that it will be easier to create/maintain distribution packages down the line as there is more of an established ecosystem and guides for setuptools I guess.

Regarding the "default" way to install: I suspect that most users will just go with what is baked into their distribution. In case of debian and derivatives, alot and its dependencies comes installable via apt install alot, done. In particular the current (v 1.18.0-4.1+b2 on testing as of now) gpg library comes packaged as python3-gpg, and is simply pulled in.

The issue with the dependencies only comes in when you want to set up a development environment, for which it makes sense to use venv and setuptools or whatever we end up suggesting. Updating the docs to reflect what currently works best sounds ok to me. In debian's case that's apt install python3-gpg + venv with --system-site-packages + pip install -e for me, but we can simply list docs for other distributions.

PS: One alternative that we did not discuss is that we could offer a container (docker, appimage) as default installation/distribution method. Then only the advanced installation instructions would be concerned with the hairy details of the gpg dependency.

Not a fan on suggesting that to users. As part of the CI that makes sense of course

PS: I see that github still does not interpret markdown in email responses :-)

[lucc]

I left the pycodestyle section in setup.cfg and I found a note in CONTRIBUTING.md that developers should use pycodestyle to check their code. Should we add CI for that or is that to strict?

[lucc]

@pazz I created a milestone "release 0.11" to mark the issues that I would like to see fixed/included in the release you are planning to do. Do you have a timeline for the release? Do you have some more issues that are important to include?

[lucc]

It seems that dependabot does not support pyproject + setuptools alone (docs). I think that this bot would be valuable so maybe we should look into adding a pipfile.lock or requirements.txt. But I am still unclear if they are a setuptools thing or a pip thing.

With poetry we had dependabot support because of the poetry lock file.

could you comment on the above and also where we'd need to keep track of dependencies?

Sorry I did not answer that yet: Currently we keep track of our dependencies in pyproject.toml and in the docs (text for humans). We also keep track of the dependencies in flake.nix but I have pushed a branch nix-pyproject where nix can read them from the pyproject.toml file (nearly automatically). With poetry we we had a similar situation that nix could read the poetry.lock file with only minor adjustments.

[pazz]

Good point about dependabot.

I am not sure about the different kinds of dependencies but this seems relevant: https://stackoverflow.com/questions/74508024/is-requirements-txt-still-needed-when-using-pyproject-toml

Would you nix script also be able to extract deps only declared in requirements.txt? Should be use that on top of the pyproject.toml file?

[lucc]

requirements.txt is a pip thing and does not replace pyproject.toml so even if we add that nix could still parse the toml file.

I found out that both setuptools and poetry both could support git repos as dependencies:

• https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#direct-url-dependencies
• https://python-poetry.org/docs/dependency-specification/#git-dependencies

That could help us with the gpg>1.10 problem from #1630 . But we have to figure out how to get to the correct subdirectory of the upstream gpgme repo at https://dev.gnupg.org/source/gpgme.git .

[pazz]

Quoting Lucas Hoffmann (2024-08-06 21:05:51)

requirements.txt is a pip thing and does not replace pyproject.toml so even if we add that nix could still parse the toml file.

From what I understand, requirements.txt is basically to set up a reproducible development environment whereas the other files list dependencies for packaging.

Seen this way it may make sense to add the requirements.txt but I don't think we have to have this. If you think it is beneficial to have this kind if "lock" file that can be semiautomatically updated by dependabot then sure, let's add it. The downside I see is that it again causes two independent list of dependencies that have to be maintained.

I found out that both setuptools and poetry both could support git repos as dependencies:

• https://setuptools.pypa.io/en/latest/userguide/dependency_management.html# direct-url-dependencies • https://python-poetry.org/docs/dependency-specification/#git-dependencies

That could help us with the gpg>1.10 problem from #1630 . But we have to figure out how to get to the correct subdirectory of the upstream gpgme repo at https: //dev.gnupg.org/source/gpgme.git .

Not sure if that is a good idea: it will mean that down the line, package maintainers will have to amend that manually and bend it so that the gpg dep is fulfilled by other system packages. Always depending on upstream branches may be too aggressive.

I think the best way forward would be to kick that old package off the pypi, and to have someone (ideally the gpgme people) upload their current version as part of the CI. Unfortunately I was not able to even find the contact details of the person owning the pypi repo.

[lucc]

I found out that both setuptools and poetry both could support git repos as dependencies.

Not sure if that is a good idea: it will mean that down the line, package maintainers will have to amend that manually and bend it so that the gpg dep is fulfilled by other system packages. Always depending on upstream branches may be too aggressive. I think the best way forward would be to kick that old package off the pypi, and to have someone (ideally the gpgme people) upload their current version as part of the CI. Unfortunately I was not able to even find the contact details of the person owning the pypi repo.

Are you on the gpg devel mailing list? Or is somebody who is concerned with the development of alot? Then we could just post there.

I am unsure what to do about the dependency update topic. As I see it dependabot stopped caring about us since we removed the poetry.lock file: https://github.com/pazz/alot/security/dependabot

[pazz]

Are you on the gpg devel mailing list? Or is somebody who is concerned with the development of alot? Then we could just post there.

I am not on their mailing list and it seems that they've closed the signup for the issue tracker/dev board https://dev.gnupg.org/T5247 There are two threads on there raising the issue about the outdated pypi package and the main (gpg) dev seems to just respond dismissively and ignores the issue. I suspect firing up a message to that list may help, just to say that it'd improve the situation if that pypi package simply wasn't there.

I am unsure what to do about the dependency update topic. As I see it dependabot stopped caring about us since we removed the poetry.lock file: https://github.com/pazz/alot/security/dependabot

yea. I don't mind really. I never really cared about db myself :)

[lucc]

I noticed that on https://alot.readthedocs.io/en/latest/installation.html#manual-installation poetry is still mentioned. And it seems that the last build on readthedocs failed: https://readthedocs.org/projects/alot/builds/25239607/

IIRC they updated their config format a while back.