Python ecosystem improvements (placeholder issue)

[domenkozar]

This is a placeholder for general improvements to Python packaging infrastructure in Nixpkgs.

Documentation

• [ ] why do we use propagatedBuildInputs?
• [ ] how to generate PYTHONPATH out of packages? (makePythonPath in nix, and toPythonPath in bash)
• [ ] explain a bit in more detail how python setup hook works and that packages not using buildPythonPackage need to add python to propagatedBuildInputs (https://github.com/NixOS/nixpkgs/issues/17679, https://github.com/NixOS/nixpkgs/pull/18143)
• [ ] wrapPython
• [ ] how to use virtualenv with Nix? (https://github.com/NixOS/nixpkgs/issues/5623, https://github.com/NixOS/nixpkgs/issues/66)

Python modules

• [x] add support for modules to all python interpreters (https://github.com/NixOS/nixpkgs/issues/12333, https://github.com/NixOS/nixpkgs/pull/19309)
• [x] enable readline by default (adds 16M to the closure (https://github.com/NixOS/nixpkgs/issues/17607, https://github.com/NixOS/nixpkgs/pull/19309)
• [x] enable ncurses by default (https://github.com/NixOS/nixpkgs/issues/492#issuecomment-194841210, https://github.com/NixOS/nixpkgs/pull/19309)
• [x] enable sqlite by default (https://github.com/NixOS/nixpkgs/issues/492#issuecomment-194841210, https://github.com/NixOS/nixpkgs/pull/19309)
• [ ] tkinter.tix doesn't yet function with $ nix-shell -p 'python.withPackages(ps: [ps.tkinter])' --run "python -c 'import Tix; Tix.Tk()'". https://github.com/NixOS/nixpkgs/pull/21643#pullrequestreview-15183437

Build functions

• [ ] join python and buildPythonPackage setup-hooks
• [x] namePrefix should be default empty and prefixed with pythonX.X- only inside python-packages.nix (https://github.com/NixOS/nixpkgs/pull/13124)
• [ ] refactor buildPythonPackage into python.mkDerivation (similar to cabal.mkDerivation)
• [x] helpers for python software that doesn't use distutils/setuptools (https://github.com/NixOS/nixpkgs/pull/18143)
• [ ] convert all mkDerivation using distutils to buildPythonPackage
• [x] convert all buildPythonPackage not using distutils to mkDerivation Typically buildPythonPackage or buildPythonApplication with format = "other"; should be used. See the docs.
• [ ] replace magicalSedExpression (https://github.com/NixOS/nixpkgs/issues/11168)
• [x] buildPythonPackage shouldn't use postFixup (https://github.com/NixOS/nixpkgs/issues/9204, https://github.com/NixOS/nixpkgs/commit/85a87f5155f66dbdfcdb9869f4bd4f2a10fe8e2c)

Packages

• [x] modernize packages in python-modules folder to be either a module or an application (https://github.com/NixOS/nixpkgs/issues/1468)
• [x] pypi2nix: a tool to generate Nix expressions for Python packages.
• [x] fixed-point combinator for python-packages.nix (https://github.com/NixOS/nixpkgs/pull/17428)
• [ ] autogenerated data with urls to and hashes for all PyPI packages (https://github.com/NixOS/nixpkgs/pull/16005, https://github.com/NixOS/nixpkgs/issues/15480).

Tests

• [ ] it should be easier to run different test runners (setup.py / nosetests / py.test) (https://github.com/NixOS/nixpkgs/pull/11715)
• [x] hook for testing importing modules (https://github.com/NixOS/nixpkgs/commit/f7e28bf5d8181926e600a222cb70180519d09726)
• [ ] import all modules after building (#3821, #14849)
• [ ] reduce closure sizes by not installing tests (https://github.com/NixOS/nixpkgs/issues/12379)

Issues to resolve

• [ ] Depend on a Python application without adding the libraries to PYTHONPATH (https://github.com/NixOS/nixpkgs/issues/11423)
• [x] Tox (https://github.com/NixOS/nixpkgs/issues/15224)
• [x] virtualenv should use pythonFull (#7134, https://github.com/NixOS/nixpkgs/pull/19309)
• [ ] unify Python interpreters (together with pypy) (https://github.com/NixOS/nixpkgs/pull/17339)
• [ ] Nix replacement for ctypes.util.find_library (https://github.com/NixOS/nixpkgs/issues/7307)
• [ ] setuptools namespaces are broken (give collisions) (https://github.com/NixOS/nixpkgs/issues/2412, https://github.com/NixOS/nixpkgs/issues/1826)
• [ ] data_files upstream issue (https://github.com/NixOS/nixpkgs/issues/4968)
• [ ] zc.recipe.egg is broken
• [ ] Circular dependencies
• [ ] pypy PYTHONPATH is incomplete (https://github.com/NixOS/nixpkgs/issues/12948)
• [x] Python on Nix is not compatible with manylinux1 (https://github.com/NixOS/nixpkgs/issues/18484, https://github.com/NixOS/nixpkgs/issues/71935)

[domenkozar]

cc @chaoflow @garbas @cillianderoiste @offlinehacker @matejc (anyone else)?

[domenkozar]

cc @rach

[shlevy]

Do you think it makes sense to make separate issues for more of these?

[domenkozar]

For some todos, if more then one line is needed to describe what's needed to be done (and if some discussion is needed).

[kevinburke]

Readline support would be very nice. Currently arrow keys in the nix python interpreter don't work.

[domenkozar]

@kevinburke could you describe exact steps to reproduce that? Stable or unstable channel?

[domenkozar]

Updated TODOs to match current state in nixpkgs.

[domenkozar]

@kevinburke once #4495 is merged, using pythonFull will mean readline support. For now you can use python and specify python.modules.readline as buildInput for your expressions.

[joelmo]

I think this could be added under issues to resolve: Import test for all derivations that add packages to site-packages folder. It would then be easier to detect if packages propagate all other packages needed for the basic import.

The test itself is very simple (python -c 'import matplotlib, pkg1, pkg2'), but it requires the correct environment to be set up after the build.

[domenkozar]

See #3821

[jagajaga]

P-p-p-ping :)

[datakurre]

I'm in need of "improved" solution for managing a lot of in-house packages in setup, where we have normal python package development and custom nix channel.

I'd prefer to have default.nix in each package repository to easily get development environments for each package. Yet, I would not like to manage duplicate buildPythonPackage-derivation in our in-house channel repository.

Have you ever considered somehow supporting explicit release.nix / default.nix in Python source packages? Could it be any more dangerous than what setup.py already is?

For my own usage, I might simply check if the package has default.nix, expect it to have buildPythonPackage derivation and call that.

[FRidh]

You could keep a default.nix file in the repo, and use src = ./.; , and then fetch the repo and import it or use callPackage. However, you will then import from another derivation, and this is currently not supported on Hydra.

[RonnyPfannschmidt]

@datakurre have you considered always having your custom channel, and each default.nix overriding the src attribute of a derivation with the path ./.

[datakurre]

@FRidh @RonnyPfannschmidt Thanks for those ideas!

I was aware that Hydra-compatibility is an issue (didn't know that limitation of derivation not being able to call derivation though), so I probably should try @RonnyPfannschmidt's approach at first. Adding new packages and adding new dependencies for developed package would be a bit inconvenient, but for existing packages it would make default.nix look less scary (for non-nixers).

I hope to end up with something that I could document as an example workflow with nix.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[OliverEvans96]

I think https://github.com/NixOS/nixpkgs/issues/91916 should be added to this list.