Error building with requirements.txt containing git dependency

[scompt]

Cool project!

I'm trying to use to build my project that has some dependencies that are pull directly from git, similar to described in this blog post.

This is the error I'm getting

``` Generating python environment... If you run this the first time, the python package index and dependency graph (~200MB) need to be downloaded. Please stay patient! unpacking 'https://github.com/DavHau/pypi-deps-db/tarball/c75c0817d61e26427fe3f7b2a82730e3f3edd62f'... warning: dumping very large path (> 256 MiB); this may run out of memory these derivations will be built: /nix/store/5szwmgq119r43aavc8bk80nv41y5byv6-nixpkgs-py-pkgs-json.drv /nix/store/xf7cwshg11lgilpvfnvgarrx677rmcw3-builder.pl.drv /nix/store/fza0yqj2p9lxnjp0pzfdwlyllxh3872c-python3-3.7.6-env.drv /nix/store/p31z3m2gxjd24bvwprn3l602n76k6xdw-mach_nix_file.drv these paths will be fetched (8.09 MiB download, 50.77 MiB unpacked): /nix/store/rndy89km7gi0kybip1mri51243n6rdd2-perl-5.30.1 building '/nix/store/5szwmgq119r43aavc8bk80nv41y5byv6-nixpkgs-py-pkgs-json.drv'... building '/nix/store/xf7cwshg11lgilpvfnvgarrx677rmcw3-builder.pl.drv'... copying path '/nix/store/rndy89km7gi0kybip1mri51243n6rdd2-perl-5.30.1' from 'https://cache.nixos.org'... building '/nix/store/fza0yqj2p9lxnjp0pzfdwlyllxh3872c-python3-3.7.6-env.drv'... created 257 symlinks in user environment building '/nix/store/p31z3m2gxjd24bvwprn3l602n76k6xdw-mach_nix_file.drv'... Traceback (most recent call last): File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 90, in __init__ req = REQUIREMENT.parseString(requirement_string) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString raise exc File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString loc, tokens = self._parse( instring, 0 ) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache loc,tokens = self.parseImpl( instring, preloc, doActions ) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl loc, exprtokens = e._parse( instring, loc, doActions ) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache loc,tokens = self.parseImpl( instring, preloc, doActions ) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 3205, in parseImpl raise ParseException(instring, loc, self.errmsg, self) pkg_resources._vendor.pyparsing.ParseException: Expected stringEnd (at char 3), (line:1, col:4) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3101, in __init__ super(Requirement, self).__init__(requirement_string) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__ requirement_string[e.loc:e.loc + 8])) pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'+git://g'" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nix/store/l5y5h21hjj4q94157yr02cpd0xg6ir18-site-packages/mach_nix/generate.py", line 54, in main() File "/nix/store/l5y5h21hjj4q94157yr02cpd0xg6ir18-site-packages/mach_nix/generate.py", line 48, in main expr = generator.generate(reqs) File "/nix/store/l5y5h21hjj4q94157yr02cpd0xg6ir18-site-packages/mach_nix/generators/overlay_generator.py", line 38, in generate prefer_nixpkgs=self.prefer_nixpkgs File "/nix/store/l5y5h21hjj4q94157yr02cpd0xg6ir18-site-packages/mach_nix/resolver/resolvelib_resolver.py", line 64, in resolve result = resolvelib.Resolver(Provider(self.nixpkgs, self.deps_provider), reporter).resolve(reqs, max_rounds=1000) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/resolvelib/resolvers.py", line 413, in resolve state = resolution.resolve(requirements, max_rounds=max_rounds) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/resolvelib/resolvers.py", line 278, in resolve for r in requirements: File "/nix/store/l5y5h21hjj4q94157yr02cpd0xg6ir18-site-packages/mach_nix/requirements.py", line 47, in filter_reqs_by_eval_marker for req in reqs: File "/nix/store/l5y5h21hjj4q94157yr02cpd0xg6ir18-site-packages/mach_nix/requirements.py", line 63, in parse_reqs reqs = list(pkg_resources.parse_requirements(strs)) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3094, in parse_requirements yield Requirement(line) File "/nix/store/zy4w84wypjwnin0bj340gkklpsffzx52-python3-3.7.6-env/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3103, in __init__ raise RequirementParseError(str(e)) pkg_resources.RequirementParseError: Invalid requirement, parse error at "'+git://g'" builder for '/nix/store/p31z3m2gxjd24bvwprn3l602n76k6xdw-mach_nix_file.drv' failed with exit code 1 error: build of '/nix/store/p31z3m2gxjd24bvwprn3l602n76k6xdw-mach_nix_file.drv' failed Traceback (most recent call last): File "/nix/store/vmccxiyv3p2qriraq4hsz6rnhbggyjlr-python3.7-mach-nix-2.0.0/bin/.mach-nix-wrapped", line 9, in sys.exit(main()) File "/nix/store/vmccxiyv3p2qriraq4hsz6rnhbggyjlr-python3.7-mach-nix-2.0.0/lib/python3.7/site-packages/mach_nix/run.py", line 134, in main gen(args) File "/nix/store/vmccxiyv3p2qriraq4hsz6rnhbggyjlr-python3.7-mach-nix-2.0.0/lib/python3.7/site-packages/mach_nix/run.py", line 28, in gen print(proc.stderr.decode(), file=sys.stderr) AttributeError: 'NoneType' object has no attribute 'decode' ```

Is this something that's not supported now? Or do I need to dig into the dependency repository for a potential bug in the package there? There's a possibility my problem is a duplicate of #19 or #23, but I'm not sure.

[DavHau]

Hey, Thanks for your report! You are getting a InvalidRequirement: Invalid requirement, parse error at "'+git://g'". No need to dig deeper. This format is just not supported.

In general it would be possible to extend the parser to allow this format. But there are a few problems with this. When pulling source code in nix, we always want to provide a hash for integrity reasons. If you'd like to include arbitrary github projects, then you would need to include the exact commit hash for the version you'd like to fetch. The format used in the blog post you mentioned, allows for including the commit hash, but doesn't enforce it. Therefore not sure if it makes sense to implement that format if it wouldn't work in half of the cases.

Right now the best way to include a python package from github would be to manually add the package to you environmant like demonstrated in this example: https://github.com/DavHau/mach-nix/issues/23#issuecomment-633915141, which is quite ugly actually. Hope i will soon find the time to implement https://github.com/DavHau/mach-nix/issues/19, which should improve this situation

[scompt]

That makes sense. Thanks! :+1:

I'll go ahead and close this issue and try my luck with the example in #23.

[hkailahi]

New user here. This tool is wonderful, thanks for creating it!

Just ran into this when following the README flow: running nix-shell ./env with an environment generated via mach-nix env ./env -r requirements.txt with a git dependency in my input requirments.txt).

I was ultimately able to resolve it after a few attempts following this repo's ./examples.md: adding myGithubDep = mach-nix.buildPythonPackage { src = builtins.fetchGit{ .. } } let-var to the generated python.nix.

I think it would be worth failing earlier in the process, or failing informatively with a link to the example for adding the packageExtras (if not populating automatically). I nearly gave up before finding this issue.

[DavHau]

That makes sense. It would be good if we did a check on the requirements format and then raise an error earlier when there are some things inside that we don't and give some recommendations.