[pip-freeze] Error: no `requirements-dev.txt`

[augustebaum]

Hi,

I'm trying out dream2nix with little experience with Nix. I have a requirements.txt file and would like to make a flake out of it. However, when I build or try to enter a shell based on the flake, I get the following error:

❯ nix shell 'github:augustebaum/dream2nix-test'
error: getting status of '/nix/store/p6q0w6pd8w08v1pbwiq8aakq03mwksf2-source/requirements-dev.txt': No such file or directory
(use '--show-trace' to show detailed location information)

I couldn't find anything about this on the "10-minute" tutorial or elsewhere; is this normal? If so, can anyone tell me what I'm doing wrong? Link to the repo: https://github.com/augustebaum/dream2nix-test.

Thanks in advance for your support! Quite excited about dream2nix apart from that :)

[con-f-use]

Did you git add requirements-dev.txt? I don't know if that file is supposed to be generated by something, or if one is supposed to populate it with their dev shell dependencies (they might be considered separate from the normal requirements.txt?). Anyways, if git doesn't know about a file, that file isn't known to flakes either. The message kinda gives that away, because it says, that nix looks for the file in /nix/store/p6q0w6pd8w08v1pbwiq8aakq03mwksf2-source/, an absolute path in the nix store, rather than in ./ or any relative path. That is because flakes want to be pure by default. Even using environment variables, e.g. something like NIXPKGS_ALLOW_BROKEN=1 requires you to flake ... --impure.

Btw. I'm also new to dream2nix and your blog-post summarizes my first experiences. I think @DavHau and the authors should take notes here, if they want a good user experience. The documentation and video presentation, leaves people scratching their heads a lot. Don't get me wrong, I understand that this is a work in progress and I'm thankful for all the work you put in, but the python experience with nix is frustrating enough without wading through a million unfinished tools that market themselves as (mostly) working just to find out, they simply don't or just in incredibly narrow circumstances.

The next thing I tried was echo '-r requirements.txt' > requirements-dev.txt; git add requirements-dev.txt, which brought me one step further:

error: builder for '/nix/store/v5sci0ivybkplvwz31lxlrjzr8rkni46-python3.10-default.drv' failed with exit code 1;
       last 10 log lines:
       > unpacking source archive /nix/store/bvxk2w39inbmyvjag6w9w7b50bk5q6kp-9hvdaaxgks5hqyhfk0vpnmgnvarjjz1j-source/
       > source root is 9hvdaaxgks5hqyhfk0vpnmgnvarjjz1j-source
       > setting SOURCE_DATE_EPOCH to timestamp 315619200 of file 9hvdaaxgks5hqyhfk0vpnmgnvarjjz1j-source/requirements.txt
       > patching sources
       > configuring
       > no configure script, doing nothing
       > building
       > WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
       > ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
       > 

That, in my opinion is the second user-interface problem here. I just want a reproducible shell with the requirements from requirements.txt. I do not want to generate a whole project. Why does it force me to create a whole structured package? And why does it accept an empty pyproject.toml without error, but in the resulting shell environment nothing is installed? mach-nix used to be able to just convert a requirements.txt into nix. If there isn't at least feature parity between the two, and #387 seems to indicate there isn't, then mach-nix shouldn't be unmaintained and it shouldn't direct people to dream2nix.

Okay, back to topic, next I skimmed really quickly through the source code and came to the conclusion, that maybe extraSetupDeps = [ "-r source/requirements.txt" ] might be a dirty hack to get dream2nix do install something from a requirements.txt. But I couldn't figure out where to specify that option and it got late and I got frustrated. Bottom line it would be nice not to advertise unfinished tools and give them well styled documentation, time will be wasted.

[con-f-use]

Update: I did manage do get it to do something:

# flake.nix
{
  inputs = { dream2nix.url = "github:nix-community/dream2nix"; };

  outputs = { self, dream2nix, } @ inp: dream2nix.lib.makeFlakeOutputs {
    systems = ["x86_64-linux"];
    config.projectRoot = ./.;
    source = ./.;
    projects = ./projects.toml;
  };
}

# projects.toml
[main]
name = "main"
relPath = ""
subsystem = "python"
translator = "pip"

[main.subsystemInfo]
pythonAttr = "python3"
extraSetupDeps = [ "-r source/requirements.txt" ]
pythonVersion = "3.9"

So the trick is to use the impure "pip" translator and extraSetupDeps. In addition to these files, I had to commit an empty pyprojects.toml and my requirements.txt. Then I can run nix build './#resolveImpure' && ./result/bin/resolve, which does a lot of piping for each of my dependencies. I'd even think it's a super-linear amount of pip calls. Then it fails for one of them. :cry:

Traceback (most recent call last):
  File "/nix/store/wnrvv6wbvalyar02n1z1d32n4ljdmvdc-generate-dream-lock.py", line 97, in <module>
    main()
  File "/nix/store/wnrvv6wbvalyar02n1z1d32n4ljdmvdc-generate-dream-lock.py", line 32, in main
    with urllib.request.urlopen(f"https://pypi.org/pypi/{pname}/json") as f:
  File "/nix/store/himpixm3rv2ld9imvsl74xqc7yyg1ndm-python3-3.9.14/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/nix/store/himpixm3rv2ld9imvsl74xqc7yyg1ndm-python3-3.9.14/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/nix/store/himpixm3rv2ld9imvsl74xqc7yyg1ndm-python3-3.9.14/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/nix/store/himpixm3rv2ld9imvsl74xqc7yyg1ndm-python3-3.9.14/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/nix/store/himpixm3rv2ld9imvsl74xqc7yyg1ndm-python3-3.9.14/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/nix/store/himpixm3rv2ld9imvsl74xqc7yyg1ndm-python3-3.9.14/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

I think that error is because the requirement comes from a custom index --index-url http://... and generate-dream-lock.py hardcodes pypi.org (also in l. 44). But even if it did not, the approach is fundamentally flawed.

So I think it's time to give up. But that might just work for your requirements.txt @augustebaum

[augustebaum]

Thanks for the detailed answer! So yeah I forgot to git add requirements-dev.txt. ^^'

Now that I've added it I'm faced with a common error when using nix on MacOS:

error: Package ‘glibc-2.35-163’ in
/nix/store/4lbmf3pbrsb0a5l4713mal64w8hp6na9-source/pkgs/development/libraries/glibc/default.nix:152
is not supported on ‘x86_64-darwin’, refusing to evaluate.

[AkechiShiro]

Just tried to do this for requirements.txt and I ran into this error, any idea why ? @con-f-use :

❯ nix develop
error:
       … in the left operand of the update (//) operator

         at «string»:56:13:

           55|             # This is shadowed in the next //
           56|             // sourceInfo
             |             ^
           57|             // {

       … while evaluating the attribute 'lib.makeFlakeOutputs'

         at /nix/store/ja8rybccb91c8qxjp8vih60byfdw6ygg-source/flake.nix:63:5:

           62|
           63|     lib = import ./lib {
             |     ^
           64|       dream2nix = inputs.self;

       error: attribute 'makeFlakeOutputs' missing

       at /nix/store/ailwni1cibwv583p9v33n250smv0k0vy-source/flake.nix:5:41:

            4|
            5|   outputs = { self, dream2nix, } @ inp: dream2nix.lib.makeFlakeOutputs {
             |                                         ^
            6|     systems = ["x86_64-linux"];