Newer nixpkgs or mach-nix break Python environment/mach-nix is unmaintained

[daneov]

@mirrexagon: I'd like to reporting that nix-shell shells/esp32-idf.nix does not seem to work

Steps performed:

git clone https://github.com/mirrexagon/nixpkgs-esp-dev.git
cd nixpkgs-esp-dev
nix-shell shells/esp32-idf.nix

Output:

The Package 'pathlib' (build: None) is not available from any of the selected providers ['nixpkgs', 'sdist', 'wheel']
 for the selected python version
The required package might just not (yet) be part of the dependency DB currently used.
The DB can be updated by specifying 'pypiDataRev' when importing mach-nix.
For examples see: https://github.com/DavHau/mach-nix/blob/master/examples.md
If it still doesn't work, there might have bene an error while building the DB.
Please open an issue at: https://github.com/DavHau/mach-nix/issues/new

error: builder for '/nix/store/87bvff7492zapj66bmxviyp3li08w1mx-mach_nix_file.drv' failed with exit code 1;
       last 9 log lines:
       >
       > The Package 'pathlib' (build: None) is not available from any of the selected providers ['nixpkgs', 'sdist', 'wheel']
       >  for the selected python version
       > The required package might just not (yet) be part of the dependency DB currently used.
       > The DB can be updated by specifying 'pypiDataRev' when importing mach-nix.
       > For examples see: https://github.com/DavHau/mach-nix/blob/master/examples.md
       > If it still doesn't work, there might have bene an error while building the DB.
       > Please open an issue at: https://github.com/DavHau/mach-nix/issues/new

Environment information:

❯ cat /etc/os-release
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"
BUILD_ID="22.11.466.596a8e828c5"
DOCUMENTATION_URL="https://nixos.org/learn.html"
HOME_URL="https://nixos.org/"
ID=nixos
LOGO="nix-snowflake"
NAME=NixOS
PRETTY_NAME="NixOS 22.11 (Raccoon)"
SUPPORT_END="2023-06-30"
SUPPORT_URL="https://nixos.org/community.html"
VERSION="22.11 (Raccoon)"
VERSION_CODENAME=raccoon
VERSION_ID="22.11"

❯ nix --version
nix (Nix) 2.11.0

I tried changing the overlay.nix by upgrading the db, but I must be doing something wrong:

diff --git a/overlay.nix b/overlay.nix
index dc4d3ae..26c8656 100644
--- a/overlay.nix
+++ b/overlay.nix
@@ -4,13 +4,13 @@ let
   mach-nix-src = prev.fetchFromGitHub {
     owner = "DavHau";
     repo = "mach-nix";
-    rev = "98d001727542bb6142d0ab554fc30bd591b07c73";
-    hash = "sha256-SXrwF/KPz8McBN8kN+HTfGphE1hiRSr1mtXSVjPJr8o=";
+    rev = "b003df0a7fb91e8e65247e0b07a774466665ae9b";
+    hash = "sha256-BRz30Xg/g535oRJA3xEcXf0KM6GTJPugt2lgaom3D6g=";
   };

   mach-nix = import mach-nix-src {
-    pypiDataRev = "1d17587404960e2e9fd0fd7e514b0bbc52abcdfd";
-    pypiDataSha256 = "sha256:078i0af4s1la5cafq958wfk8as711qlf81ngrg0xq0wys7ainig1";
+    pypiDataRev = "982b6cdf6552fb9296e1ade29cf65a2818cbbd6b";
+    pypiDataSha256 = "sha256:166y0li0namv6a8ik8qq79ibck4w74x0wgypn9r7sqbb2wvcvcf3";
     pkgs = final;
   };
 in

[mirrexagon]

I haven't been able to get updates to nixpkgs or mach-nix to work either. Also see #10.

I'm not sure what to do in general with the IDF Python dependencies. The current options I can think of are:

• Keep trying to make mach-nix and later nixpkgs work.
• Try to switch to dream2nix, which is noted as the successor to mach-nix, but Python support doesn't seem to be there yet.
• Switch to a virtual-env-based solution. That's what I did to get InvokeAI working, posting that shell.nix unchanged for reference, most likely won't work for IDF as-is but can serve as a starting point for exploring this:

{ pkgs ? import <nixpkgs> { } }:

with pkgs;

mkShell rec {
  name = "invoke-ai-pip-shell";
  venvDir = "invokeai";

  buildInputs = [
    python310Packages.python
    python310Packages.venvShellHook
  ];

  LD_LIBRARY_PATH = lib.makeLibraryPath [
    stdenv.cc.cc.lib
    cudaPackages.cudatoolkit
    cudaPackages.cudnn
    libGL
    glib
  ] + ":/run/opengl-driver/lib";

  postVenvCreation = ''
    unset SOURCE_DATE_EPOCH
    ln -sf environments-and-requirements/requirements-lin-cuda.txt requirements.txt
    pip install -r requirements.txt
  '';

  postShellHook = ''
    # allow pip to install wheels
    unset SOURCE_DATE_EPOCH

    export INVOKEAI_ROOT=~/invokeai

    echo 'Run: invoke.py --web --host 0.0.0.0'
  '';
}

[mirrexagon]

15 did some updates to the PyPi DB revision, but I still don't think it works with newer nixpkgs unfortunately.

[mirrexagon]

I'm making this the master issue for all the newer nixpkgs/mach-nix/Python environment problems.

[mirrexagon]

To consider: using constraints files for Python package versions - see https://github.com/mirrexagon/nixpkgs-esp-dev/pull/15#issuecomment-1604016511

The issue with this is they might change under us and won't have a stable hash.

[mirrexagon]

I manually packaged the required Python packages that weren't already in nixpkgs, instead of using mach-nix - it should work with recent nixpkgs now.