Issue building a requirements.nix for a package that has only wheels available for its latest version.

[ilyaGotfryd]

I am building a requirements.nix the latest version (v1.1.1) of https://pypi.org/project/asyncnsq/ to use in my project however since the last package that contains tar.gz package is version 0.4.5 I can only see packages up to that version.

curl https://pypi.org/simple/asyncnsq/

for details.

[seppeljordan]

Hi, yes. With the current process this is unavoidable. pypi2nix assumes that we cannot trust any wheels that are prebuilt. You can circumvent this by building directly from the upstream source code repository. Try adding the following to your requirements.txt

git+https://github.com/aohan237/asyncnsq.git@1.1.0#egg=ayncnsq

Haven't tested this though. Let me know if this worked out for you. Also you could create an issue on their site asking them to upload tar archives to pypi.

[ilyaGotfryd]

I was actually working against v1.1.0 and ended up modifying the requirements.nix derivation output as follows:

"asyncnsq" = python.mkDerivation {
      format = "wheel";
      name = "asyncnsq-1.1.0";
      src = pkgs.fetchurl {
        url = "https://files.pythonhosted.org/packages/2d/85/873d884cb15cc12b1774ee52b35b52ad3dfd61ea17404f9017e8c26cafb7/asyncnsq-1.1.0-py3-none-any.whl";
        sha256 = "f59d6a54036ae2aae63ee448af1d44b1d9815be2577a985ba7e44a3b79cd9702";
      };
      doCheck = commonDoCheck;
      buildInputs = commonBuildInputs ++ [ ];
      propagatedBuildInputs = [
        self."python-snappy"
      ];
      meta = with pkgs.stdenv.lib; {
        homepage = "https://github.com/aohan237/asyncnsq";
        license = licenses.mit;
        description = "asyncio async/await nsq support";
      };
    };

I copied wheel URL and sha256 by hand and added format="wheel"; to derivation and it worked in my package. Is it not reasonable to permit prefetching wheel directly if one is available?

[seppeljordan]

What you are proposing could work out, but for that I'd have to implement a mechanism to detect if the wheel available is suitable for the current platform. Also this would mean that derivations produced on one system can not be used on a different platform, e.g. arm vs intel, linux vs windows.

[ilyaGotfryd]

So workflow would potentially be Wheel if one is available for the platform, otherwise, look for a tarball of that version and build else fail if an unreachable version for that platform is requested? Is it something that pip already taking care of internally? Also if you are interested in addressing this where would be the correct insertion point for this logic? I was looking for something to possibly work on for Hacktober.

[seppeljordan]

Hi, I would be grateful support on developing this and will gladly help you as far as I can. But I am not super eager to introduce this into the code base. Here is some thoughts that I had on fetching wheels:

• To ensure compatibility to other platforms than the build platform we need to fall back to source distributions if we do not find an appropriate wheel
• We cannot use any wheels with c extensions since the wheels downloaded might be linked against /usr/lib
• Building from source should be the default in my opinion since most people expect that from nix
• In my opinion it makes more sense to submit issues asking to release a source distribution than working around that (Open source!)
• I expect it to be harder patching a wheel (if necessary) than a source distribution. This could lead to complications if we ever decide to host a central place for overrides. (We did that once at garbas/nixpkgs-python)

That said, if you want to explore the this issue in the code, let me point you to a couple of source files:

• src/pypi2nix/stage2.py contains the machinery to metadata from pypi
• src/pypi2nix/pypi.py contains the interface to pypi
• src/pypi2nix/wheel.py contains our model of a wheel

[ilyaGotfryd]

Thank you for the write-up. This is a very helpful insight. I have done Python for a few years, but I am absolutely new to NixOS so I have barely any exposure to the underlying philosophy. In light of what you wrote, I wonder if I could take the following stance in implementation.

• When reading the desired package version from requirements.txt or command line and comparing them PyPI records the system will check both wheel and source package versions available
• If only wheel version available, check if that version is available for the target system
• If yes still issue a version error, but indicate that a wheel is available for the desired version
• Also indicate that utility can be run with --allow-wheel-packages to address this problem

This way pypi2nix remains true to form, however, allows for exceptions. I am not sure if I am misreading the Python community intentions, but it appears to me that wheel is the way of the future.

[seppeljordan]

Hi, I agree with all your 4 points except the third one. You are right in that we should let the user know that there are newer versions than the downloaded one but throwing an error seems a little extreme. I suggest that the standard behavior can be a warning just before pypi2nix finished execution. Also exclude pre versions since most larger software projects publish pre-releases. Maybe we could have a flags called --allow-old-source-distribution/--no-allow-old-source-distribution where --allow-old-souce-distribution is the default. I guess implementing a warning for the user about old source distributions could be done with reasonable effort right now.

[seppeljordan]

BTW: The primary repository of this program is here.

[ilyaGotfryd]

Is there a way to move that issue there? I was trying to find an answer to my question and ended up in this repo. 🤓

[seppeljordan]

Unfortunately I cannot move this issue to another user.