add a nix flake

[teto]

when I packaged alot for poetry, I did it within a ubuntu container and I realized it was not that easy to setup (I've updated instructions in the doc in that same PR). Nix is a package manager aiming at reproducibility and it excels in it.

With nix installed one can run nix run 'github:teto/alot/flake' to test the PR (though it is currently failing). Likewise nix develop 'github:teto/alot/flake' will give you a development environment with all dependencies available.

Note: this is easy to maintain out of tree. The 2 flake.* files can be moved in a contrib/ folder to not crowd the root directory.

[lucc]

I would welcome a flake in the alot repo.

I did set up a wrapping flake repo a while ago with a gtihub action to fetch the latest alot: https://github.com/lucc/alot-flake. Having the flake in the main alot repo would simplify a few things in my opinion:

• the workflow in lucc/alot-flake is regularly deactivated by github because the repo does not change for a while, so it would be nice to get rid of the extra repo and the workflow
• I have to maintain the building logic outside of the alot repo resulting in divergent states (right now my flake does not handle the new pyproject file I think)
• the flake in the alot main repo could be used in CI which would fix some of my trouble in #1617

But

• the flake is mostly for the benefit of people using the nix package manager which is not python specific (but can be used in ci)
• we can still maintain the flake in a separate repo (like I already do), so that's no problem if @pazz is against it.

[pazz]

I am not against this at all and happy to facilitate whatever makes usage and development easier. If you continue to maintain this then all the better :) P

Should I just merge this then?

Quoting Lucas Hoffmann (2023-07-28 10:43:11)

I would welcome a flake in the alot repo.

I did set up a wrapping flake repo a while ago with a gtihub action to fetch the latest alot: https://github.com/lucc/alot-flake. Having the flake in the main alot repo would simplify a few things in my opinion:

• the workflow in lucc/alot-flake is regularly deactivated by github because the repo does not change for a while, so it would be nice to get rid of the extra repo and the workflow • I have to maintain the building logic outside of the alot repo resulting in divergent states (right now my flake does not handle the new pyproject file I think) • the flake in the alot main repo could be used in CI which would fix some of my trouble in #1617

But

• the flake is mostly for the benefit of people using the nix package manager which is not python specific (but can be used in ci) • we can still maintain the flake in a separate repo (like I already do), so that's no problem if @pazz is against it.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.*Message ID: <pazz/alot/pull/ @.***>

[lucc]

From my point of view we can merge as soon as @teto has fixed draft state.

[teto]

Should I just merge this then?

dont it's not ready yet, I wanted to get your sentiment first but I've seen @lucc active on nixpkgs before so that's a positive point. I will have a look at @lucc's flake. This PR "autogenerates" the nix code from the pyproject.toml to avoid duplicating the code dependencies but it still needs some overrides/tweaks.

[lucc]

@teto our CI was defuct for quite some time and it might be that some tests are broken. You might have to disable some test cases explicitly. The alot derivation in nixpkgs does it, my flake also does it.

[teto]

@lucc so it should be ok to test. Let me know if nix run github:teto/alot/flake wokrs for you.

We can improve the flake afterwards. I will try to finish https://github.com/pazz/alot/pull/1603 and will see what is missing then

[lucc]

@teto do we even need a dedicated devShell? When I run nix develop .\#packages.x86_64-linux.default I have a python with all dependencies (I can run python -m alot in that shell). So how about we get rid of the devShell stuff and merge this?

[teto]

Agreed, let's merge we can refine later.

[pazz]

I should have reported this here earlier:

I just tried to install via poetry on a new system (debian testing) and it fails with gpg related errors. Did we overlook any dependencies?

poetry install --no-root

Installing dependencies from lock file    

Package operations: 1 install, 0 updates, 0 removals

  • Installing gpg (1.10.0): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel

  Could not find gpgme-config.  Please install the libgpgme development package.

  at ~/.venv/lib/python3.11/site-packages/poetry/installation/chef.py:147 in _prepare
      143│ 
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│ 
      146│             if error is not None:
    → 147│                 raise error from None
      148│ 
      149│             return path
      150│ 
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with gpg (1.10.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "gpg (==1.10.0)"'.

pip wheel --use-pep517 "gpg (==1.10.0)"
Collecting gpg==1.10.0
  Using cached gpg-1.10.0.tar.gz (39 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      Could not find gpgme-config.  Please install the libgpgme development package.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I'm on debian 12 and successfully installed the dependencies indicated in this commit using

sudo apt get install python3-setuptools python3-magic python3-configobj python3-notmuch python3-urwid python3-urwidtrees python3-gpg python3-twisted python3-dev swig

There has been some issues/discussions about gpgme-config missing in current versions of libgpgme-dev and other projects have worked around this.

Is this simply a debian testing issue with gpgme and pip or do we need to take action?

[teto]

It's possible I missed libgpgme-dev in the list of dependencies. On nixos it appears in the python package:

nix-locate bin/gpgme-config
libsForQt5.qgpgme.dev                             5,393 x /nix/store/a33nb2h0145dc0yhwih1dzxdik7b574d-gpgme-1.20.0-dev/bin/gpgme-config
python310Packages.gpgme.dev                       5,389 x /nix/store/i98bybk6d6ci3rcmmvhw7pw4mypif57q-gpgme-1.20.0-dev/bin/gpgme-config
python311Packages.gpgme.dev                       5,389 x /nix/store/jjg0kr6z221mqkmm0rz5cg5da9frf88s-gpgme-1.20.0-dev/bin/gpgme-config
gpgme.dev                                         5,390 x /nix/store/h7x6m11chlxbd5i9jd4saxkv1jbn4yf8-gpgme-1.20.0-dev/bin/gpgme-config

maybe apt has a python3-gpgme on top of python3-gpg ?

[pazz]

I have installed all those packages where available, but think this binary simply has been removed from debian (and Arch?): https://github.com/pspdev/pspdev/issues/44

Quoting Matthieu Coudron (2023-08-29 10:23:54)

It's possible I missed libgpgme-dev in the list of dependencies. On nixos it appears in the python package:

nix-locate bin/gpgme-config libsForQt5.qgpgme.dev 5,393 x /nix/store/a33nb2h0145dc0yhwih1dzxdik7b574d-gpgme-1.20.0-dev/bin/gpgme-config python310Packages.gpgme.dev 5,389 x /nix/store/i98bybk6d6ci3rcmmvhw7pw4mypif57q-gpgme-1.20.0-dev/bin/gpgme-config python311Packages.gpgme.dev 5,389 x /nix/store/jjg0kr6z221mqkmm0rz5cg5da9frf88s-gpgme-1.20.0-dev/bin/gpgme-config gpgme.dev 5,390 x /nix/store/h7x6m11chlxbd5i9jd4saxkv1jbn4yf8-gpgme-1.20.0-dev/bin/gpgme-config

maybe apt has a python3-gpgme on top of python3-gpg ?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.*Message ID: <pazz/alot/pull/ @.***>

[lucc]

@pazz I am confused: is your comment about the nix flake or about the poetry integration which is from #1621?

[pazz]

@lucc I am confused as to where the difference is tbh. Fact is, I cannot install the development version (used to be ./setup.py develop) using poetry nor pip.

I am also a little confused as to where we declare dependencies now. I hope not three times? I know this complaint comes late, apologies.

By the way, If I change pyproject.toml like so, then I can install via pip install -e .

diff --git a/pyproject.toml b/pyproject.toml
index 52feecf6..d7cfddb3 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -29,7 +29,7 @@ urwidtrees = ">=1.0.3"
 twisted = ">=18.4.0"
 python-magic = "*"
 configobj = ">=4.7.0"
-gpg = "*"
+gnupg = "*"

 [tool.poetry.dev-dependencies]
 pycodestyle = "*"

On debian testing, there should be no need to install gpgme wia pip or poetry because the documented dependencies (in alot's install docs) include python3-gpg (nowadays python-gpg) which comes with the python bindings to gpg as used in alot.

[lucc]

The core difference between poetry and nix to my understanding is:

• scope:
  • poetry is a package manager for python packages
  • nix is a general package manager
• files:
  • poetry uses pyproject.toml and poetry.lock
  • nix uses flake.nix and flake.lock

I think the fuzzyness comes from the fact that we use the poetry integration for nix, i.e we let nix read the poetry.lock file in order to read and build the dependencies of alot. (If you check flake.lock -- its quite short -- you can see that it does not list any dependencies of alot)

So where do we list dependencies:

• we list our python deps in pyproject.toml/poetry.lock in a machine readable format
• we list our non python deps in flake.nix in a format readable by the nix package manager
• we (apparently) list all deps in the user manual for debian/ubuntu/fedora users

The patch for pyproject.toml looks like it should also fix this override in flake.nix: https://github.com/pazz/alot/blob/67ca49a/flake.nix#L32

[pazz]

Thank you, this clears up a lot for me. Still, the current status is kind of broken on debian, not due to us:

alot uses the gpg module, which are the python bindings for direct calls to libgpg. This needs to be compiled during the installation via pip install gpg, which fails on debian testing and derivates because updated libgpg development packages come withou the gpgme-config tool.

I can install alot via pip install -e . or poetry install --no-root if I change the dependency as in my patch above. But then it fails to run due to a missing import og gpg.

poetry run alot
Warning: 'alot' is an entry point defined in pyproject.toml, but it's not installed as a script. You may get improper `sys.argv[0]`.

The support to run uninstalled scripts will be removed in a future release.

Run `poetry install` to resolve and get rid of this message.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/patrick/projects/alot/alot/__main__.py", line 10, in <module>
    from alot.settings.const import settings
  File "/home/patrick/projects/alot/alot/settings/const.py", line 4, in <module>
    from .manager import SettingsManager
  File "/home/patrick/projects/alot/alot/settings/manager.py", line 17, in <module>
    from ..utils import configobj as checks
  File "/home/patrick/projects/alot/alot/utils/configobj.py", line 14, in <module>
    from .. import crypto
  File "/home/patrick/projects/alot/alot/crypto.py", line 5, in <module>
    import gpg
ModuleNotFoundError: No module named 'gpg'

Not surprising.

Now, I do have a working gpg python module installed via my package manager. Somehow this system-wide package is not picked up by pip nor poetry though..

As side-node: the line in the flake.lock file you linked to looks as if we already do some mocking around the gpg module. And it looks to me that maybe it'd be a good idea to use the gpgme module instead of gpg because that's actively being maintained ?

[lucc]

To my understanding poetry does everything in a venv so it will not pick up your system python stuff.

About my comment: I was mixing up the strings gpg, gnupg and gpgme so not sure if that comment is any good.

[lucc]

I can reproduce the breakage on debian with this docker file:

FROM debian
RUN apt -yq update && apt install -yq python3-poetry libnotmuch-dev libmagic1
# used by the gnupg python package?
RUN apt install -yq libgpgme-dev
# not picked up by poetry?
RUN apt install -yq python3-gpg
# fails with "Could not find gpgme-config.  Please install the libgpgme development package."
RUN poetry install

What is the relation between the different gpg modules for python? I thought that https://pypi.org/project/gpg/ was from the upstream people from https://www.gnupg.org, at least that's what they link as their homepage.

[teto]

https://github.com/pazz/alot/pull/1624#issuecomment-1699233793 couldn't be any better it's perfect.

Somehow this system-wide package is not picked up by pip nor poetry though..

You can disable the venv created by poetry https://github.com/python-poetry/poetry/issues/110#issuecomment-389591605 but that feels wrong.

I had tested the poetry script in an ubuntu container, not debian. Do we need gpgme-config for email signature validation ? How is it used ?

[lucc]

Do we need gpgme-config for email signature validation ? How is it used ?

gpgme-config is used by the python gpg package during installation. The error message stems from poetry -> pip -> gpg package.

The gpg package is used for all things crypo in alot.