[Docs] Lack of fully fledge example with actually working dev env and build setup

[aMOPel]

Describe the issue

What do I mean with actually working developer environment?

A shell that:

1. contains the python interpreter installed from nix wrapped with the pinned python packages. The problem is, that some python packages require correctly linked dynamic C libraries to work. Simply importing python and python packages from nixpkgs separately won't work. There is a work around with nix-ld, however as I understood it, that is why python.withPackages exists, which handles this wrapping for you. poetry2nix uses python.withPackages in mkPoetryEnv.

2. doesn't require the developer to send the developed package through the nix-store again, after every change. If the derivation of the developed package itself is put into the nix-shell, you have to rebuild the nix-shell to reflect source code changes, which is seriously hindering workflow.

3. enables python tooling (e.g. the LSP). The python tooling needs to be able to find the imported python packages.

There are some examples, but none are satisfying all 3 requirements:

• the template uses mkPoetryApplication for the developer shell, which violates 2..
• the readme example uses mkPoetryEnv but tooling doesn't find the imported libs, which violates 3..
• the how-to-guide blogpost simply uses poetry itself to manage the imported python packages, and nix just to manage the python interpreter and poetry. This violates 1..

Also they all do things completely different, which is unnecessarily confusing for users.

Also beside the minimalist template example there are no other whole project examples.

My mediocre non-flake solution

I hope this can serve other beginners as a more or less usable template.

Setup

This assumes usage of niv to pin nixpkgs and poetry2nix.

./nix/release.nix

let
  sources = import ./sources.nix;
  pkgs = import sources.nixpkgs { };
  poetry2nix = import sources.poetry2nix { inherit pkgs; };
  my_app = pkgs.callPackage ./build.nix { inherit pkgs poetry2nix; };
in
{
  inherit my_app;
}

./nix/build.nix

{ pkgs
, poetry2nix
}:
let
  python = pkgs.python311;
  projectDir = ./.;

  # just to read name and version from pyproject.toml
  pyProject = (poetry2nix.mkPoetryPackages {
    inherit python projectDir;
  }).pyProject;
  name = pyProject.tool.poetry.name;
  version = pyProject.tool.poetry.version;

  # NOTE: add non python packages, needed at runtime, here
  runtimePackages = with pkgs; [
  ];

  # NOTE: add non python packages, only needed for development, here
  devPackages = with pkgs; [
    poetry
  ];

  # NOTE: add build time dependencies for python packages here, when confronted with
  # ModuleNotFoundError: No module named '...'
  # see https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md
  pypkgs-build-requirements = {
    # <package> = [ "<missing build tools>" ];
  };
  p2n-overrides = poetry2nix.defaultPoetryOverrides.extend (final: prev:
    (builtins.mapAttrs
      (package: build-requirements:
        (builtins.getAttr package prev).overridePythonAttrs (old: {
          buildInputs = (old.buildInputs or [ ])
            ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg)
            build-requirements);
        })
      )
      pypkgs-build-requirements)
  );

  build = poetry2nix.mkPoetryApplication {
    inherit python projectDir;
    # NOTE: trade off
    # "rebuild everything from scratch, which can take forever"
    # vs
    # "pull wheels from pypi, when available and accept supply chain attack risks"
    # also necessary, when having errors with `setuptools-rust`
    # preferWheels = true;
    overrides = p2n-overrides;
  };

  dev-env = poetry2nix.mkPoetryEnv {
    inherit python projectDir;
    # NOTE: see above
    # preferWheels = true;
    editablePackageSources = {
      "${name}" = ./src;
    };
    overrides = p2n-overrides;
  };

  shell = pkgs.mkShell {
    # this is the important bit to enable python tooling and thus satisfy all 3 requirements
    # WARNING: the problem is that the python version is hardcoded in this path
    PYTHONPATH = "${dev-env}/lib/python3.11/site-packages";

    packages = [
      dev-env
    ]
    ++ runtimePackages
    ++ devPackages;
  };

  image = pkgs.dockerTools.buildLayeredImage {
    contents = [
      # for debugging the container
      # pkgs.busybox

      build
    ] ++ runtimePackages;
    inherit name;
    tag = version;
    # maxLayers = 100;
    config = {
      Cmd = [
        "/bin/${name}"
      ];
    };
  };
in
{ inherit build shell image; }

./shell.nix

let
  packages = import ./nix/release.nix;
in
packages.my_app.shell

Usage

The first time:

1. nix-shell -p poetry
2. Setup pyproject.toml and poetry.lock: poetry init

Manipulate python dependencies:

1. enter dev shell: nix-shell (or use direnv)
2. poetry <verb> <package>@<version>
3. re enter shell: exit followed by nix-shell

The dev shell has the pinned python version, poetry version and all python packages from the poetry lock. The python packages are also in PYTHONPATH, which should be picked up by your dev tooling.

Build app:

• nix-build -A build

Build image:

• nix-build -A image

Bump version:

• change version pyproject.toml, this will change it in the build and the image.

[drawnwren]

Here's mine, I stole your PYTHONPATH. The only thing not working for me is editablePackages for dev tools (pyright complains about relative imports unless I poetry shell).

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication mkPoetryEnv defaultPoetryOverrides;
            pypkgs-build-requirements = {
              onecache = [ "setuptools" ];
              pyright = [ "setuptools" ];
              aiosonic = [ "setuptools" ];
              inflector = [ "setuptools" ];
              flupy = [ "setuptools" ];
              alembic-utils = [ "setuptools" ];
              opentelemetry-exporter-otlp  = [ "cython" ];
              uvloop = [ "cython" pkgs.libuv ];
              # pendulum = [ "maturin" ];
            };
            p2n-overrides = defaultPoetryOverrides.extend (final: prev:
              builtins.mapAttrs (package: build-requirements:
                (builtins.getAttr package prev).overridePythonAttrs (old: {
                  buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
                })
              ) pypkgs-build-requirements
            );
      in
      {

        # Use this shell for developing your app.
        devShells.default = let 

            pkg_deps = with pkgs; [
              ruff
              ruff-lsp
              docker-compose
              docker
              poetry
              postgresql_16
              cassandra
            ];
            envShell = mkPoetryEnv {
              projectDir = ./A;
              python = pkgs.python311;
              overrides = p2n-overrides;
              preferWheels = true;
              extraPackages = ( ps: with ps; [
                ruff-lsp
                pylint
                pip
                six
              ]);
              editablePackageSources = {
                B = ./B;
              };
            }; 
          in envShell.env.overrideAttrs (old: {
            buildInputs = pkg_deps;
            LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
            PYTHONPATH = "${envShell}/lib/python3.11/site-packages";
          });
      });
}

[drawnwren]

One problem with the PYTHONPATH for dev approach is that it will ruin some of your cli packages. awscli2 for example imports urllib3 and getting the wrong one will render the package unusable.

[aMOPel]

You probably have to append to the pythonpath instead of overriding it, to prevent that problem. Good point though.

PYTHONPATH = "${envShell}/lib/python3.11/site-packages":$PYTHONPATH;

[aMOPel]

The only thing not working for me is editablePackages for dev tools (pyright complains about relative imports unless I poetry shell).

What exactly is the scenario? You change file names and then imports break? Don't quite get it.

[aMOPel]

You're adding ruff-lsp twice

And I wonder, why do you need to set the LD env var?

[drawnwren]

So, the LD_LIBRARY_PATH solves this error ValueError: the greenlet library is required to use this function. libstdc++.so.6: cannot open shared object file: No such file or directory for me.

editablePackages just doesn't pick up the imports in my LSP. All of the pyproject.toml imports in my current directory resolve fine but local dependencies in other projects on disk don't resolve.

I don't think appending $PYTHONPATH solves the cli problem because it's finding the wrong version not missing the import completely.