Review Nix-setup

[datakurre]

@johbo Since also you've been using pip2nix to help Python development at work, would you be interested in and able to review and discuss about the pip2nix based generic/reusable Nix environment in this project. I try hard to support best practices for both traditional Python development and Nix packaging. I'm planning to blog about it in detail later:

• ./setup.py is mostly empty and everything, including setup_requires, install_requires, test_requires, is in declarative ./setup.cfg (supported by current setuptools)
• ./requirements.txt includes everything from setup.cfg, but also QA tools like flake8 used indevelopment
• ./nix-support/requirements.nix is pip2nix -generated from requirements.txt
• ./nix-support/default.nix includes generic helpers build from the above
  • setup is setup.cfg parsed into nix language
  • pythonPackages is the final pythonPackages extended with requirements.nix + overrides.nix
  • env builds python.withPackages to provide interpreter with requirements.txt e.g. for PyCharm
  • shell is convenient shell shortcut for the above env usable with nix-shell nix-support -A shell
  • pip2nix provides shell with pip2nix to allow easy update of requirements.nix
• ./default.nix is generic buildPythonPackage derivation for building the developed python Package (and to provide development shell for that; it uses parsed setup.cfg to populate name, buildInputs and propagatedBuildInputs).
• ./release.nix provides optional Hydra build targets

That's a lot of Nix-files, but everything besides docs-target in ./release.nix should be generic and usable in other projects without modifications.

[datakurre]

To prove the reusability of my approach, I made another iteration into branch

https://github.com/collective/sphinxcontrib-httpexample/tree/setup-nix

And exported reusable Nix expressions into

https://github.com/datakurre/setup.nix

Next I'm going to write README for setup.nix project.

With this iteration python package repository only needs to have setup.nix and requirements.nix. requirements.nix is the file generated and maintained using pip2nix. setup.nix includes overrides and calls setup.nix-project to make all setup.nix-derivations available:

• nix-build setup.nix -A build builds the package with minimal buildInputs

• nix-shell setup.nix -A develop enters package development shell with all packages in requirements.nix in its buildInputs

• nix-build setup.nix -A env builds python interpreter with all package in requirements.nix (but not the package itself)

• nix-build setup.nix -A install builds python interpreter only with the package itself

• nix-build setup.nix -A sdist builds sdist for the package

• nix-build setup.nix -A bdist_wheel builds wheel for the package

• nix-shell setup.nix -A pip2nix --run "pip2nix generate -r requirements.txt --output=requirements.nix" updates requirements.nix

Naming is hard. I tried to follow existing names from normal setup.py commands. env is just env, because Nix-created env is not really venv.

In addition setup.nix also exposes a version of pythonPackages with all packages in requirements.nix.

[johbo]

Hi, yes, although I am quite busy this week, cannot promise that I will manage to follow up in a timely manner 😬

I've got some stuff put up here: https://github.com/johbo/pip2nix-generated. During the NixCon hackathon I got some time to make use of pip-compile to have a cleaner process. I like the result so far.

[datakurre]

@johbo Thank you. Don’t worry about this week then. I continue to work with this with a couple of new in-house packages this week and hopefully get ”setup.nix”-project and reasoning behind it polished for easier review, and keep you up posted.

My main motivations are 1. make use of nix look less scary for our other developers, and 2. minimize work needed to maintain nix compatibility for individual Python packages (we don’t manage custom Nix channel or have custom nix packages collection, but each package/repo should be buildable on its own).

[datakurre]

I've now merged "setup.nix"-approach and continue to document it with examples at https://github.com/datakurre/setup.nix

[datakurre]

@johbo I close this issue in favour of this https://github.com/datakurre/setup.nix/issues/1

I’d still appreciate if you had change to take a look on it next week. I upgraded a few in-house projects to it with good results. I also create a pull for pip2nix to fix the issues I had with namespaced packages and pip2nix.

Thank you again for your time!