search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.95k stars 13.97k forks source link

Python packages installing absolute paths to $out/lib/python-xxx/site-packages #23438

Open bennofs opened 7 years ago

bennofs commented 7 years ago

There's some python packages in nixpkgs that fail due to the use of the wheel python build system. Here's one case that I manually fixed: https://github.com/aquynh/capstone/pull/783.

The reason for this seems to be that with wheels, paths in in data_files in the setup.py are always interpreted relative to the SITE_PACKAGES directory, even if they start with a /. Some packages, however, try to manually construct an absolute path, which is how we end up with double absolute paths.

Here's a list of affected packages (nix-locate 'site-packages/nix/store'):

I haven't checked it this is a problem for the function of the packages (it could still be harmless). Perhaps we should fix these before the release, like #23253? (/cc @globin)

nico202 commented 7 years ago

Don't know why so much noise, however let me know: 1) if it's really fixed and 2) if you prefer not to use realpath

FRidh commented 7 years ago

Let me guess, they determine whether we're installing in a "virtualenv" and want to keep the data_files outside of it?

That seems to be the case in Cloud Init https://git.launchpad.net/cloud-init/tree/setup.py?id=0.7.9#n97

FRidh commented 7 years ago

Would it be "bad" if we fix sys.real_prefix to return the same value as sys.prefix? Of course, that would "break" the test these packages use to determine whether they'e in a virtualenv. Note that sys.real_prefix was removed in Python 3.x

bennofs commented 7 years ago

@FRidh I don't think this only happens with virtualenv. Look at the setup.py of the pkg I linked (capstone) for example

bennofs commented 7 years ago

@FRidh I'm not even sure if there is a way to write packages such that they behave the same under wheel and non-wheel method, without explicitly checking if building a wheel.

FRidh commented 7 years ago

I don't seem to understand the issue here. If there are differences between wheel and non-wheel, then that is something for wheel, pip or setuptools to fix. If packages do something weird in their setup.py, then that's a bug on their side.

bennofs commented 7 years ago

@FRdih yes I agree with that. Notifying upstream about the issue is a valid solution here, but as it stands, such packages are often subtly broken. Should we distribute broken packages in nixpkgs?

bennofs commented 7 years ago

I pinged the maintainers of the packages to give you a chance to check if this impacts your package and then take appropriate actions. I'm not using all of these packages, but in my experience, this particular issue often leads to strange problems that packages cannot find their data files later. So feel free to ignore this if everything is fine.

7c6f434c commented 7 years ago

Great. For getmail it was only about documentation. setup.py spends some effort to build the absolute path, and the comments complain that you cannot install using a relative path…

Short-circuited the simplest-to-kill part of all that. Thanks for noticing.

mmahut commented 5 years ago

Any news on this issue?

bennofs commented 5 years ago

@mmahut this is still an issue, I updated the first post with the current list of affected packages

FRidh commented 5 years ago

This is actually a duplicate of https://github.com/NixOS/nixpkgs/issues/4968.

FRidh commented 5 years ago

Actually, let's use this one instead as it is more up to date.

worldofpeace commented 5 years ago

Huh, I could've swore we fixed this for ulauncher. I'll look into it.

ciil commented 5 years ago

It's on my radar. I'll look into it over the weekend.

seqizz commented 4 years ago 
nix-shell -p 'python3.withPackages(ps : with ps; [ uvicorn fastapi ])'   
these derivations will be built:
  /nix/store/b0x3ghq2p0hhpdnv5a21c0vmrq73krxd-python3-3.7.5-env.drv
building '/nix/store/b0x3ghq2p0hhpdnv5a21c0vmrq73krxd-python3-3.7.5-env.drv'...
collision between `/nix/store/m8nps45w28fpzzvh7jqcm5mmq6njgi0y-python3.7-starlette-0.12.4/LICENSE.md' and `/nix/store/wvv320lnlcv2yxzw3khpjsaxkmnzrgbr-python3.7-uvicorn-0.9.0/LICENSE.md'
builder for '/nix/store/b0x3ghq2p0hhpdnv5a21c0vmrq73krxd-python3-3.7.5-env.drv' failed with exit code 25
error: build of '/nix/store/b0x3ghq2p0hhpdnv5a21c0vmrq73krxd-python3-3.7.5-env.drv' failed

Adding two more packages :confused:

jtojnar commented 4 years ago

Posting the upstream issue here as well to make it easier to find: https://github.com/pypa/setuptools/issues/130

emilazy commented 4 years ago

It's quite easy to accidentally package something missing its runtime dependencies due to its setup.py misusing sys.prefix in this way; I ran into this with #86683 (fixed upstream in https://github.com/Nicotine-Plus/nicotine-plus/pull/166). Perhaps the Python packaging machinery should have a simple check that you haven't created a nix directory in sys.prefix so that this error would at least result in a noisy failure rather than silent breakage?

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/errno-13-when-trying-to-build-a-python-application/10741/8

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

emilazy commented 3 years ago

still bad

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info

bennofs commented 2 years ago

still bad

FRidh commented 2 years ago

Yes, it's still an upstream issue. There is no spec for dealing with these files when using wheels. https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853

alexvorobiev commented 2 years ago

Adding two more

$ nix-shell -p 'python3.withPackages(ps : with ps; [ requirements-parser cyclonedx-python-lib ])'

building '/nix/store/6mbwfc4578cyplh3lb8g4fg9vh19hch4-python3-3.9.12-env.drv'...
error: collision between `/nix/store/fc4frxp2mwa5xcr8hrkcmn8pk8knwilf-python3.9-cyclonedx-python-lib-2.3.0/lib/python3.9/site-packages/LICENSE' and `/nix/store/c34666c339bgnqparpcn0ssppgjprcmk-python3.9-requirements-parser-0.5.0/lib/python3.9/site-packages/LICENSE'
error: builder for '/nix/store/6mbwfc4578cyplh3lb8g4fg9vh19hch4-python3-3.9.12-env.drv' failed with exit code 25;
       last 1 log lines:
       > error: collision between `/nix/store/fc4frxp2mwa5xcr8hrkcmn8pk8knwilf-python3.9-cyclonedx-python-lib-2.3.0/lib/python3.9/site-packages/LICENSE' and `/nix/store/c34666c339bgnqparpcn0ssppgjprcmk-python3.9-requirements-parser-0.5.0/lib/python3.9/site-packages/LICENSE'
       For full logs, run 'nix log /nix/store/6mbwfc4578cyplh3lb8g4fg9vh19hch4-python3-3.9.12-env.drv'.
senorsmile commented 1 year ago

3 more packages that seem to have the same issue:

Background: I was trying to install the pypi docker package and running into the issue that it wasn't found when trying to import. Eventually ended up here: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#environment-defined-in-etcnixosconfigurationnix-environment-defined-in-etcnixosconfigurationnix. Following those instructions, found that numpy and toolz weren't being found either, but finally noticed the collision message, which led me to this page's issue.

As an not so convenient work around, I am able to run the python containing the right environment from: 

warning: collision between `/nix/store/wp58vaq76dahy2af1hnbqqld1cif7pkj-python3-3.10.8-env/bin/python' and `/nix/store/zdba9frlxj2ba8ca095win3nphsiiqhb-python3-3.10.8/bin/python'

Which leads to success: 

# /nix/store/wp58vaq76dahy2af1hnbqqld1cif7pkj-python3-3.10.8-env/bin/python
Python 3.10.8 (main, Oct 11 2022, 11:35:05) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import docker
senorsmile commented 1 year ago

I believe that the issue I'm seeing is different than this issue, although all of the other documentation led me to believe otherwise. I determine that the above work when I remove the explicit installation of python3. So, all three of the packages appear fine.