search

cript0nauta / pynixify

Nix expression generator for Python projects
GNU General Public License v3.0
66 stars 11 forks source link

Modify local clone of Nixpkgs, for upstreaming packages #50

Open catern opened 4 years ago

catern commented 4 years ago

First off, awesome project, this makes it much easier to make Python packages which are upstreamable for Nixpkgs! I just used pynixify to generate and make a PR for https://github.com/NixOS/nixpkgs/pull/97155 and it worked great!

I think things could be even easier with a few extra features - maybe in a different mode or something.

  1. Put any new packages in pkgs/development/python-modules/
  2. Modify python-packages.nix to add the new packages

Those are the main things, but as a stretch goal, it would be also be cool if there were options to:

  1. List the current user as maintainer in the meta section of the packages (maybe could even autodetect from username?)
  2. Don't add doCheck = false; since I'll always want to try building the package with tests at least once
  3. Autodetect the license from PyPI and fill in that section.

I think if pynixify had 1 and 2, pynixify could be endorsed in the manual as the encouraged way to create a new Python package for Nixpkgs. It would cut down on a lot of PR review effort spent on badly-formatted manually created trivial packages.

cript0nauta commented 4 years ago

First off, awesome project, this makes it much easier to make Python packages which are upstreamable for Nixpkgs! I just used pynixify to generate and make a PR for NixOS/nixpkgs#97155 and it worked great!

I'm glad you found it useful :)

  1. Put any new packages in pkgs/development/python-modules/
  2. Modify python-packages.nix to add the new packages

I think the first item would be easy to develop, but the second one would be harder, since it would require to parse an existing Nix file and modifying it. I'm not sure how would I implement it in a robust way. Do you know any project that automatically modifies nixpkgs files?

List the current user as maintainer in the meta section of the packages (maybe could even autodetect from username?)

This would be useful! I should think in a way that is useful both for individuals and for team projects.

Autodetect the license from PyPI and fill in that section

This would also be nice, although making the mapping from PyPI licenses to nixpkgs licenses can be hard. When I thought in adding this feature, I noted that there wasn't a consistent way of specifying licenses in PyPI packages, but i might be wrong.

Don't add doCheck = false; since I'll always want to try building the package with tests at least once

pynixify has some command line options to load test requirements (see pynixify --help for more information). When it succeeds to find test requirements, it won't add doCheck = false. This doesn't work very well because there isn't a standardized to specify the test command and dependencies, but that's a limitation of the Python ecosystem. Have you tried using this options?

catern commented 4 years ago

I think the first item would be easy to develop, but the second one would be harder, since it would require to parse an existing Nix file and modifying it. I'm not sure how would I implement it in a robust way. Do you know any project that automatically modifies nixpkgs files?

nixpkgs-pytools has python-package-init which modifies python-packages.nix. It seems to use a fairly basic method without truly parsing Nix code: https://github.com/nix-community/nixpkgs-pytools/blob/master/nixpkgs_pytools/output.py#L54

That's a bit hacky, but honestly it seems pretty reasonable to me. If it ever results in problems it could always be improved later.

This would also be nice, although making the mapping from PyPI licenses to nixpkgs licenses can be hard. When I thought in adding this feature, I noted that there wasn't a consistent way of specifying licenses in PyPI packages, but i might be wrong.

What about the PyPI license classifiers? pdoc3 has it for example: https://pypi.org/project/pdoc3/

pynixify has some command line options to load test requirements (see pynixify --help for more information). When it succeeds to find test requirements, it won't add doCheck = false. This doesn't work very well because there isn't a standardized to specify the test command and dependencies, but that's a limitation of the Python ecosystem. Have you tried using this options?

Ah, I didn't notice this! That is very helpful. Maybe this option could be the default?

Although, perhaps I'm not using it right, but pynixify pdoc3 --tests pdoc3 still add doCheck = false though it didn't print an error messages. As it happens this package has no additional test requirements.

Actually, for upstreamable Nix packages, maybe doCheck = false shouldn't be added automatically even if test requirement discovery failed. A packager should always be encouraged to try to run the tests at least once.

cript0nauta commented 4 years ago

nixpkgs-pytools has python-package-init which modifies python-packages.nix. It seems to use a fairly basic method without truly parsing Nix code: https://github.com/nix-community/nixpkgs-pytools/blob/master/nixpkgs_pytools/output.py#L54

That's a bit hacky, but honestly it seems pretty reasonable to me. If it ever results in problems it could always be improved later.

Thanks for sharing this code! This does exactly what I'd want for this feature, so I'll consider using part of this code in pynixify. It certainly looks hacky, but for now it could be an experimental feature, only enabled when using an option like --into-nixpkgs-repo=~/nixpkgs.

Ah, I didn't notice this! That is very helpful. Maybe this option could be the default?

In an ideal world, this would be the default behavior. Unfortunately, some packages have lots of test dependencies, some of them may be written in C and fail to be built when using pynixify. Since adding the testing dependencies only benefits a subset of users (nixpkgs contributors), I prefer the feature to be off by default so it doesn't negatively affect regular users.

Although, perhaps I'm not using it right, but pynixify pdoc3 --tests pdoc3 still add doCheck = false though it didn't print an error messages. As it happens this package has no additional test requirements.

This problem occurs because it only removes the doCheck = false when the packages has some testing dependency: https://github.com/cript0nauta/pynixify/blob/main/pynixify/expression_builder.py#L75. This is an incorrect assumption for some packages, so I think I should fix this.