search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.37k stars 14.33k forks source link

Convert remaining python2 applications over to python3 #101964

Closed jonringer closed 2 years ago

jonringer commented 4 years ago

In https://github.com/NixOS/nixpkgs/pull/101929 we have many important programs (e.g. cachix) still using python2 in their builds. This list doesn't constitute python2 applications, but rather packages which have python2Packages.cryptography somewhere in their dependency graph. So these will be affected when python2Packages.cryptography does get marked as vulnerable.

This issue to track the conversion process over to python3, packages still needing to be converted are listed below. This list isn't exhaustive, just those that use the soon-to-be-marked-vulnerable pythonPackages.cryptography:

Finding the dependency

For most dependencies, it should be pretty obvious where python2 comes from, for more "difficult" packages. You may need to do some digging.

nix-tree + nix-instantiate

you can run nix-shell -p nix-tree --run "nix-tree $(nix-instantiate default.nix -A <package>) to get the entire build dependency tree, then search for the cryptography package, and then you should be able to trace which dependencies are introducing it.

nix why-depends

alternatively, you can use nix why-depends nix why-depends --all -f default.nix <package> python2Packages.cryptography can also be used, however, this will require you to re-build the package, which may take more time than parsing the dependency tree above

SuperSandro2000 commented 4 years ago 
 python3.7-aria2p
 python3.7-pygccxml
 python3.7-pyside
 python3.7-pyside-shiboken
 python3.7-pyside-tools
 python3.8-aria2p
 python3.8-pygccxml
 python3.8-pyside
 python3.8-pyside-shiboken
 python3.8-pyside-tools

Why are those in this list? Do they have a python2 variant?

jonringer commented 4 years ago

not entirely sure. It could be that some library they use, uses python2 to do something like generate docs. So they aren't directly using python2 packages.

primeos commented 4 years ago

Why are those in this list? Do they have a python2 variant?

I assume those are mostly cases of Python 3 packages that depend on some non-Python package that in turn depends on some Python 2 package (e.g. python3Packages.aria2p -> aria2 -> ... -> python27Packages.cryptography).

The following can help to navigate the dependency trees more efficiently (/ to search for the Python 2 cryptography and then h to navigate up in the dependency tree):

$ nix-tree $(nix-instantiate -A python3Packages.aria2p)

Should I try to write a script (or does someone already know/have one) to ping the maintainers of the affected packages?

SuperSandro2000 commented 4 years ago

then h to navigate up in the dependency tree):

The root is on the left and then use vim style key bindings or arrow keys.

domenkozar commented 4 years ago

cachix fixed in 59c53bc62e1e25015bd8deedd7252ee5037314b3

jonringer commented 4 years ago

I added some directions on how to locate cryptography in a dependency tree

jonringer commented 4 years ago

oh, nix-tree is way easier to navigate

jonringer commented 4 years ago

I tried packaging asciidoc-py3, but everything they do has xml imports to web urls.... :(

mkenigs commented 4 years ago

Looks like amazon-glacier-cmd-interface has python2 code: https://github.com/uskudnik/amazon-glacier-cmd-interface/blob/9f28132f9872e1aad9e956e5613b976504e930c8/glacier/glacier.py#L40

SuperSandro2000 commented 4 years ago

The last real commit is over 6 years ago https://github.com/uskudnik/amazon-glacier-cmd-interface/commits/master. Maybe think about removing it.

mkenigs commented 4 years ago

@SuperSandro2000 removed it

jtojnar commented 4 years ago

@jonringer our XML packages include findXMLCatalogs hook which should make them available when you add them as dependencies.

prusnak commented 4 years ago
freezeboy commented 4 years ago

Some packages are still only available for python2 :facepalm::

freezeboy commented 4 years ago

inspectrum is simply depending on gnuradio also in the list, so it can be removed from here. gqrx too

gdown is packaged a python*Packages independant, so there is no problem.

freezeboy commented 4 years ago

babashka doesn't seem to depend on python but it uses graalvm which expression is quite big (and not up to date, but I prefer to ping @volth or @hlolli to patch this one)

hlolli commented 4 years ago

@freezeboy when https://github.com/NixOS/nixpkgs/pull/99631 gets merged, python2x wont be a dependency of graalvm anymore.

freezeboy commented 4 years ago

opae must be updated to version 2.0.0-1

new version depends on pybind11, but there is a hacky cmake module trying to download it, I don't want to patch the cmake modules.

freezeboy commented 4 years ago

@volth maybe for the python interpreters but libraries might not get maintainance from their developers for python2 branch, so it is safer I guess to transition the maximum number of programs when possible

SuperSandro2000 commented 4 years ago

There are still maintained python2 forks (including commercially supported, for example ActiveState/cpython), it is possible to cherry-pick CVE fixes from there (if anyone concerted in security of tools which run only in buildtime sandbox)

I wouldn't do that. At some point Python 2 should just be thrown out and if software is not updated until that date it should be marked as insecure, broken or removed.

jonringer commented 4 years ago

@freezeboy I updated the packages which upstream only provides a python2 variant

freezeboy commented 4 years ago

Thank you @jonringer, I just spotted tsung also which has a pending PR to support python3 (https://github.com/processone/tsung/pull/352), but not yet packaged unfortunately

freezeboy commented 4 years ago

carddav-util also is written in python2 and the project has no activity since 2018

freezeboy commented 4 years ago

euca2ools is also written in python2 even with latest release from 2017

freezeboy commented 4 years ago

glslViewer too, even if it has more recent updates is using python2 syntax

jonringer commented 4 years ago

I'm doing glslviewer now, the python2 scripts are very simple, and able to just use 2to3 with no problem.

jonringer commented 4 years ago

@thoughtpolice @domenkozar @rvl do you mind taking a look at bumping datadog, I'm not super experience with go, and looks like they made the build process painful (api key, and a bunch of wrapped invoke tasks) https://github.com/DataDog/datadog-agent

doronbehar commented 4 years ago

@jonringer I marked some gnuradio related applications with an [x] since they are taken care of in #99685 .

jonringer commented 4 years ago

@anna328p do you mind taking a look at hercules-ci-agent

jonringer commented 4 years ago

list should be up-to-date now

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-should-stable-nixos-prioritize/9646/55

rvl commented 4 years ago

We stopped using datadog-agent and started hosting our own prometheus monitoring. The datadog-agent build process is quite painful, so it might be a bit of work to update to the latest version.

freezeboy commented 4 years ago

In addition there are 2 versions of datadog-agents in nixpkgs

jonringer commented 4 years ago

@rvl I noticed that, looks like they have a multi-language build now with cmake, go, and python https://github.com/DataDog/datadog-agent#getting-started

omasanori commented 4 years ago

Could I ask you to pin this issue, if possible? It is worth gaining publicity, as no one probably wants to maintain Python 2 until Autumn 2021.

glittershark commented 4 years ago

@jonringer https://github.com/NixOS/nixpkgs/pull/102693 gets us graal and all the related stuff

glittershark commented 4 years ago

not mx, though.

jonringer commented 4 years ago

besides graal packages, I think we are close to closing this.

dasJ commented 4 years ago

@jonringer asciidoc-full-with-plugins should be fixed (on master) with #102398

SuperSandro2000 commented 4 years ago

This PR #102919 removes Python2 from Mono.

domenkozar commented 4 years ago

I'm going to unpin this issue as I'd like to get attention on Apple+Arm issue.

jonringer commented 4 years ago

@domenkozar that's fine, this is almost complete. Will probably wrap it up over the weekend

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/python-2-7-and-3-9-and-hydra-builds/10130/4

fabianhjr commented 4 years ago

Not on the list but recoll 1.25 switched to python3 ( https://www.lesbonscomptes.com/recoll/pages/release-history.html ), recoll is currently on 1.24, #104699 upgrades it to the latest version, 1.27.

jonringer commented 4 years ago

This list doesn't constitute python2 applications, but rather packages which have python2Packages.cryptography somewhere in their dependency graph.

but getting rid of python2 completely is a long term goal, so :+1:

jonringer commented 4 years ago

most of these changes should be backported, assuming they don't cause regressions (most of them just had python2 as a build dependency)

jonringer commented 4 years ago

Thanks everyone for helping :)

FRidh commented 4 years ago

Still several left so keeping this open.

jonringer commented 4 years ago

I closed it because most of the low-hanging fruit was done.

The remaining items are not trivial to fix. Since nixpkgs is just a package repository, it's not really within our domain to have to do large fixes to upstreams to make them work. Once a stable release which uses python3 is available, we can package it. Users will just have to list the drv name as a known vulnerability in there nixpkgs config.

I'm not a big fan of leaving this issue open. If you do want to have an open issue, then i would make one per specific package; but I would say this effort is largely completed.

FRidh commented 3 years ago

Next step would be to make python an alias to python2 in aliases.nix and evaluate Nixpkgs with allowAliases = false;.