nix-community / poetry2nix

Convert poetry projects to nix automagically [maintainer=@adisbladis,@cpcloud]
MIT License
838 stars 437 forks source link

Monorepo confusion about editablePackageSources and flake.nix location #1719

Open drawnwren opened 2 months ago

drawnwren commented 2 months ago

I have a monorepo structured like so:

repo/
    services/
        a/
            pyproject.toml
         b/
            pyproject.toml

Where a imports b as editable with b = {path="../b", develop=true} in the pyproject.toml. I'm having trouble getting my nix development derivation to import b correctly into the shell.

I've tried:

  1. a/flake.nix with a derivation like:
    
        devShells.default = let 
            envShell = mkPoetryEnv {
              projectDir = self;
              overrides = p2n-overrides;
              preferWheels = true;
              editablePackageSources = {
                b = ../b;
              };
            }; 
          in
          pkgs.mkShell {
            name = "A";
            buildInputs = [ envShell ];
            #inputsFrom = [ self.packages.${system}.a ];
            packages = with pkgs; [ poetry docker-compose unzip ruff-lsp stdenv.cc.cc.lib zip awscli2 postgresql];
            LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
        };
      });
My understanding was that nix (like docker) copies the entire sub-directory structure, so it might be missing "sibling" folders (i.e. w/ same parent directory)
2. I've also tried putting `flake.nix` in `services/` with something like
    devShells.default = let 
        envShell = mkPoetryEnv {
          projectDir = ./a;
          overrides = p2n-overrides;
          preferWheels = true;
          editablePackageSources = {
            b = ./b;
            a = ./a;
          };
        }; 
      in
      pkgs.mkShell {
        name = "A";
        buildInputs = [ envShell ];
        packages = with pkgs; [ poetry docker-compose unzip ruff-lsp stdenv.cc.cc.lib zip awscli2 postgresql];
        LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
    };
  });


But I still get the `ModuleNotFoundError` for `b`.

What's the expected way for me to make this structure work?
fmnxl commented 1 week ago

What worked for me was using the submodule as a flake input:

inputs = {
  my-library = {
    url = "git+file:my-library-dir";
    flake = false;
  };
}

Then setting editablePackageSources to point to it:

outputs = { self, ..., my-library }: {
  # ...
  devShell = mkPoetryEnv {
    editablePackageSources = {
      my-library = my-library;
    };
};

And keep pyproject.toml as if importing the library from pip. No need to set path and develop = true

The downside is I have to run nix flake lock --update-input my-library every time I make a change.

But with how flakes are coupled to the git state, I don't think it'd be possible to import the library without flattening it into the parent's/monorepo git tree. As in, I don't think it's possible for flakes to link to submodules dynamically, as everything is locked by flake.lock.