mach-nix gen invalid expression git package

Hi. I have this in my requirements.txt:

git+https://github.com/discogs/python-cas-client.git#egg=cas-client

mach-nix gen fail with this message:

[nix-shell:/work]# mach-nix gen -r requirements.txt
path is '/nix/store/ahaz90hy6lins0a56mdivrd2fjj3rcb5-554d2d8aa25b6e583575459c297ec23750adb6cb'
this derivation will be built:
  /nix/store/1ra92kylck7i3lcniymkgivmgglnbqmx-mach_nix_file.drv
building '/nix/store/1ra92kylck7i3lcniymkgivmgglnbqmx-mach_nix_file.drv'...
Traceback (most recent call last):
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/distlib/markers.py", line 139, in interpret
    expr, rest = parse_marker(marker)
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/distlib/util.py", line 142, in parse_marker
    return marker(marker_string)
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/distlib/util.py", line 132, in marker
    lhs, remaining = marker_and(remaining)
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/distlib/util.py", line 121, in marker_and
    lhs, remaining = marker_expr(remaining)
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/distlib/util.py", line 108, in marker_expr
    lhs, remaining = marker_var(remaining)
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/distlib/util.py", line 76, in marker_var
    raise SyntaxError('invalid expression: %s' % remaining)
SyntaxError: invalid expression: //github.com/discogs/python-cas-client.git

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/x8sk695y4612x3znmannpi3da4y0f25y-site-packages/mach_nix/generate.py", line 110, in <module>
    main()
  File "/nix/store/x8sk695y4612x3znmannpi3da4y0f25y-site-packages/mach_nix/generate.py", line 103, in main
    do()
  File "/nix/store/x8sk695y4612x3znmannpi3da4y0f25y-site-packages/mach_nix/generate.py", line 66, in do
    expr = generator.generate(reqs)
  File "/nix/store/x8sk695y4612x3znmannpi3da4y0f25y-site-packages/mach_nix/generators/overides_generator.py", line 36, in generate
    pkgs = self.resolver.resolve(reqs)
  File "/nix/store/x8sk695y4612x3znmannpi3da4y0f25y-site-packages/mach_nix/resolver/resolvelib_resolver.py", line 55, in resolve
    result = resolvelib.Resolver(Provider(self.nixpkgs, self.deps_provider), reporter).resolve(reqs, max_rounds=1000)
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/resolvelib/resolvers.py", line 413, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/resolvelib/resolvers.py", line 278, in resolve
    for r in requirements:
  File "/nix/store/x8sk695y4612x3znmannpi3da4y0f25y-site-packages/mach_nix/requirements.py", line 57, in filter_reqs_by_eval_marker
    if distlib.markers.interpret(str(req.marker), context):
  File "/nix/store/iy0g6vf4vscdwdh8szbrjwc9ciczjpyk-python3-3.9.9-env/lib/python3.9/site-packages/distlib/markers.py", line 141, in interpret
    raise SyntaxError('Unable to interpret marker syntax: %s: %s' % (marker, e))
SyntaxError: Unable to interpret marker syntax: //github.com/discogs/python-cas-client.git: invalid expression: //github.com/discogs/python-cas-client.git
error: builder for '/nix/store/1ra92kylck7i3lcniymkgivmgglnbqmx-mach_nix_file.drv' failed with exit code 1

Hi, i needed the same feature today, so I made this function below, to map those URLs into something mach-nix can use. I don't think mach-nix can or should consume unpinned sources, so the below solution relies on the URL containing a git commit.

Nix file:

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix";
    ref = "refs/tags/3.4.0";
  }) { };

  inherit (mach-nix.nixpkgs) lib;

  # Given input requirements (str), return an attrset of 'packages' (list)
  # containing mach-nix.buildPythonPackage results from 'git+https://...'
  # entries in the requirements, and 'requirements' (str) being the remaining
  # requirements.
  splitRequirementsAndPackages = requirements:
    let
      lines = lib.splitString "\n" requirements;
      urlLines = lib.filter (lib.hasPrefix "git+https://") lines;
      nonUrlLines = lib.filter (x: !lib.hasPrefix "git+https://" x) lines;
      getUrl = pipUrl:
        lib.removePrefix "git+" (builtins.elemAt (lib.splitString "@" pipUrl) 0);
      getRev = pipUrl:
        let
          parts = builtins.elemAt (lib.splitString "#egg=" pipUrl) 0;
        in
          builtins.elemAt (lib.splitString "@" parts) 1;
      getPname = pipUrl:
        builtins.elemAt (lib.splitString "#egg=" pipUrl) 1;
      buildPackage = pipUrl:
        let
          src = builtins.fetchGit {
            url = getUrl pipUrl;
            rev = getRev pipUrl;
            allRefs = true;
          };
        in
          mach-nix.buildPythonPackage {
            pname = builtins.unsafeDiscardStringContext (getPname pipUrl);
            version = "0-${toString src.revCount}";
            # This fails to resolve in mach-nix for some reason:
            #version = "0-${toString src.revCount}-g${src.shortRev}";
            inherit src;
          };
    in
      { requirements = lib.concatStringsSep "\n" nonUrlLines;
        packages = map buildPackage urlLines;
      };

  originalRequirements = ''
    git+https://github.com/discogs/python-cas-client.git@f1efa2f49a22d43135014cb1b8d9dd3875304318#egg=cas-client
  '';

  reqsAndPkgs = splitRequirementsAndPackages originalRequirements;
in
  mach-nix.mkPython {
    requirements = reqsAndPkgs.requirements;
    packagesExtra = reqsAndPkgs.packages;
  }

Build and run:

$ nix-build ./mach-nix-issue-406.nix && ./result/bin/python3 -c "import cas_client; print(cas_client.__version__)"
trace: 
 automatically detected requirements of cas-client:0-40 22.05pre-git:
pycryptodome
requests
six
tox

/nix/store/1vp2771zk8gpr1xk8ip8rlz8x6s0y7nr-python3-3.9.9-env
1.0.0

It would be nice to have this feature integrated into mach-nix, but I'm not sure how (yet).

DavHau / mach-nix

mach-nix gen invalid expression git package #406