search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.02k stars 13.37k forks source link

python3Packages: get rid of uses of nose #326513

Open emilazy opened 1 month ago

emilazy commented 1 month ago

nose is abandoned upstream and has caused us untold headaches recently (admittedly mostly unrelated to the code itself). We’ve stopped the bleeding by getting https://github.com/NixOS/nixpkgs/pull/325968 into staging thanks to the incredible work of @jchv, but it’s still broken on master and we ought to do some tidying up so we’re not dragging this around forever and perhaps even to unblock some stuff on master while the staging cycle progresses. Many packages don’t actually need nose any more, despite us including it in our dependency lists. Active upstreams should have already moved to pytest or something else. Some packages will have patches available; if we’re really feeling generous we could send patches to upstream ourselves. Others may be completely unmaintained packages that we could think about dropping.

Please make sure to migrate packages to pyproject = true; when working on them. Additionally, if the upstream is unmaintained, there are no significant reverse dependencies in‐tree or on Wheelodex, and it requires patching or there is no particular reason to imagine we’d want to keep it, consider just dropping the package. For the same reasons we want to get rid of nose, we should take this as an opportunity to reduce future maintenance burden and drop broken, unused, or long‐obsolete packages.

@dotlambda has already done some great work on this – see https://github.com/NixOS/nixpkgs/pull/325847, https://github.com/NixOS/nixpkgs/pull/325872, https://github.com/NixOS/nixpkgs/pull/325885, and https://github.com/NixOS/nixpkgs/pull/326164 – but I thought it’d be good to have a master list of all affected packages so we can tackle it collectively. If you open a PR for one of these, please edit this issue to link it beside the relevant package(s).

This was generated from staging commit 83881cb6e9b660cbca5241380c031ab2b69d6e1e with the following awful fish one‐liner: for pkg in (rg -l '\bnose3?\b' | rg -v '/(nose[23]?|cufflinks|enocean|jaconv|annoy|hdmedians|flaky|pycron|snapshottest|ssh-mitm|colour|influxdb)/|(python|cran)-(packages|aliases)\.nix' | sed 's|.*/\([^/]*\)/\([^/]*\)$|\1|'); rg "/$pkg"'[ \n{]' pkgs/top-level/python-packages.nix | awk '{print "python3Packages." $1}'; test $pipestatus[1] = 0 || rg "/$pkg"'[ \n{]' pkgs/top-level/all-packages.nix | awk '{print $1}'; test $pipestatus[1] = 0 || echo "$pkg"; end | sort -u | sed 's/^/* [ ] `/; s/$/`/'; feel free to edit to update the list.

lavafroth commented 1 month ago

python3Packages.biopandas has dropped pynose in v0.5.1.dev1 (not yet released). We should be able to check that off once the next version releases.

cyrinux commented 1 month ago

Hi, "paperwork" package also depend on nose and raise error actually.

pyrox0 commented 1 month ago

Seems like paperwork depends on pypillowfight, which has a build error due to nose not being supported. I'm investigating now!

Edit: it also depends on fabulous, which has some more failing tests.

pyrox0 commented 1 month ago

ledger-autosync can stay broken for now, per https://github.com/egh/ledger-autosync/issues/143, upstream is working on getting a new release out, which should include all fixes for distutils, nose -> pytest, etc. I could also just update to the latest commit in the meantime if that's what's wanted.

pyrox0 commented 1 month ago

sasview, while listed here, does not depend on nose.

lavafroth commented 1 month ago

@pyrox0 sasview has an xhtml2pdf.nix besides the default.nix which depends on nose.

Sigmanificient commented 1 month ago

I am not a package maintainer of any of the mentioned packages, may I still help?

pyrox0 commented 1 month ago

@Sigmanificient yes you may! feel free to submit a PR for any listed package ^^

eclairevoyant commented 1 month ago

Yep, anyone can submit a PR for any package, as long as it follows guidelines; maintainership != exclusive ownership.

nixos-discourse commented 1 month ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/error-nose-1-3-7-not-supported-for-interpreter-python3-12/48703/40

lavafroth commented 1 month ago

We should probably mark the packages we have discovered that still use nose so that we don't have to repeat each other's homework.

emilazy commented 1 month ago

As in, the packages that would need patching to drop the dependency? Seems like a good idea; if anyone without permissions to edit the original post lists such packages here I’ll add a note to the list and hide the comment to avoid noise.

Sigmanificient commented 3 weeks ago

What should be done for trfl? It is marked as broken

eclairevoyant commented 3 weeks ago

Drop it? It hasn't seen activity in the 2 years since it was init'ed...

Sigmanificient commented 3 weeks ago

Drop it? It hasn't seen activity in the 2 years since it was init'ed...

Oh this is reassuring, I was about to do a drop pr, but I noticed that it was only marked as broken 2 month ago.

Sigmanificient commented 3 weeks ago

styra was marked as broken 8 months ago: https://github.com/NixOS/nixpkgs/commits/master/pkgs/development/python-modules/stytra/default.nix

lavafroth commented 3 weeks ago

pyrfc3339 needs patching to drop nose. Last commit was 5 years ago. Python 3.2+ appears to have support for this natively (see here). The builtin dateutil module supports parsing an RFC3339 timestamp back. Should we drop it?

emilazy commented 3 weeks ago

Yes, let’s drop it. In general, anything that’s giving us trouble during the 3.12 migration and clearly has no maintenance is most likely going to cause us more trouble in future; the same reasons we want to get rid of nose apply to most of the packages that will make getting rid of nose a pain.

Of course, there are some packages that may be too popular or too irreplaceable for dropping them to be easy to stomach, but I do think that in the cases where upstream actually does depend on nose and doesn’t seem actively maintained enough for a PR to be worthwhile, we should lean towards just dropping the package by default. Those packages are going to fall out of the active Python ecosystem anyway because you won’t be able to use them on the latest Python version with the standard package managers; in the absence of a compelling reason to make an exception, it doesn’t do much good for us to keep them on life support only for users of Nixpkgs.

It’s easy to re‐add a removed package if people complain or if something that requires it gets added to Nixpkgs, but hard for us to actively keep broken packages working. We should think of this as an opportunity to tend our garden rather than just doing a bunch of labour for dead upstreams and no real benefit. As far as making the judgement call goes, in‐tree reverse dependencies are an obvious metric to use for libraries, but I also found a site called Wheelodex that indexes reverse dependencies for PyPI projects. That can help inform the judgement calls about whether to patch or drop.

eclairevoyant commented 3 weeks ago

Do you know an easy way to build an in-tree reverse dependency list for a given package? (As in the full tree, not just one level up)

emilazy commented 3 weeks ago

Do you know an easy way to build an in-tree reverse dependency list for a given package? (As in the full tree, not just one level up)

No, I’ve just been grepping for immediate reverse dependencies and then repeating if anything looks likely to be used by anything else. I know there’s a script by @trofi that probably does a better job, and ofborg of course knows how to do it.

By the way, in the case of pyRFC3339, it looks like Certbot depends on it. Maybe that’s just a compatibility thing and it can use the standard library on newer versions, but if not we probably can’t drop it.

eclairevoyant commented 3 weeks ago

I don't think borgo knows anything about it? I'll look at trofi's script though, thanks.

emilazy commented 3 weeks ago

It calculates the number of rebuilds to tag by checking which output paths change. The ofborg-eval action output lists all of them.

emilazy commented 3 weeks ago

Maybe this will come in handy for stuff we need to patch: https://github.com/pytest-dev/nose2pytest.

eclairevoyant commented 3 weeks ago

It calculates the number of rebuilds to tag by checking which output paths change. The ofborg-eval action output lists all of them.

yes but it gets the number of changed hydraJobs (which requires editing the expression first), it doesn't query for reverse deps (afaik)

trofi commented 3 weeks ago

Using https://github.com/trofi/nixpkgs-overlays/blob/main/lib/arevdeps.nix I got this list:

$ nix-instantiate --impure --read-write-mode --eval --strict ~/.config/nixpkgs/lib/arevdeps.nix --argstr attrName python311Packages.nose -I nixpkgs=$PWD --arg verbose 2 --arg depth 2 ``` python311Packages.actdiag python311Packages.biopandas python311Packages.blockdiag python311Packages.boto python311Packages.braintree python311Packages.bx-python python311Packages.cassandra-driver python311Packages.cgroup-utils python311Packages.changefinder python311Packages.chart-studio python311Packages.dm-control python311Packages.dodgy python311Packages.envs python311Packages.flask-restful python311Packages.hkdf python311Packages.http-ece python311Packages.jsonable python311Packages.libgpuarray python311Packages.locationsharinglib python311Packages.lockfile python311Packages.mohawk python311Packages.mwtypes python311Packages.mwxml python311Packages.neo python311Packages.nose python311Packages.nose-pattern-exclude python311Packages.nose-timer python311Packages.nose-warnings-filters python311Packages.nose-xunitmp python311Packages.nwdiag python311Packages.ofxhome python311Packages.ofxtools python311Packages.para python311Packages.paramz python311Packages.paver python311Packages.pid python311Packages.pprintpp python311Packages.preggy python311Packages.premailer python311Packages.prison python311Packages.prox-tv python311Packages.ptable python311Packages.pycdio python311Packages.pycontracts python311Packages.pydy python311Packages.pyexcel-ods python311Packages.pyexcel-xls python311Packages.pygatt python311Packages.pygeoip python311Packages.pygogo python311Packages.pygtfs python311Packages.pylacrosse python311Packages.pypass python311Packages.pyquaternion python311Packages.pyrfc3339 python311Packages.pysrt python311Packages.python-etcd python311Packages.python-hglib python311Packages.python-mapnik python311Packages.pytimeparse python311Packages.pyutilib python311Packages.rauth python311Packages.safe python311Packages.sampledata python311Packages.scikit-fuzzy python311Packages.selectors2 python311Packages.seqdiag python311Packages.sphinx-rtd-dark-mode python311Packages.sqlalchemy-mixins python311Packages.sure python311Packages.svgutils python311Packages.telfhash python311Packages.tempita python311Packages.tissue python311Packages.traittypes python311Packages.unicode-slugify python311Packages.vxi11 python311Packages.webhelpers python311Packages.weboob python311Packages.xlwt ```
Sigmanificient commented 3 weeks ago

I'll try to keep a checklist up to date, based on @trofi's list:

97 items: 30.93% done, 23.71% in progress, 44 waiting moved down

emilazy commented 3 weeks ago

The list in my original issue is the relevant one. The packages present in @trofi’s list but not mine don’t depend directly on nose, so there’s nothing actionable to do with them. The reverse dependencies script is potentially useful when assessing whether to drop individual packages in my list.

emilazy commented 3 weeks ago

Ah, sorry; I forgot that arevdeps.nix only does one level of reverse dependencies. So actually it’s relevant, and maybe some of the differences between the two might be due to the imprecision of my grep script. (But e.g. aws-google-auth is missing from @trofi’s list, and it definitely references nose. Not sure what’s going on there.)

Sigmanificient commented 3 weeks ago

Boto was archived may 10, 2024

emilazy commented 3 weeks ago

Although boto is deprecated, a few things still depend on it, probably including out‐of‐tree projects. I’m not qualified to say if it’d be possible to do an immediate sweep of them, but I’d probably lean towards fixing it instead.

lavafroth commented 3 weeks ago

Seems like package weboob is a duplicate of woob?

Sigmanificient commented 3 weeks ago

webhelpers has doCheck = false due to broken code, but I cant access the source Is is on my side, or repository is gone: https://bitbucket.org/bbangert/webhelpers

lavafroth commented 3 weeks ago

@Sigmanificient the source code should still be there on Pypi but yes, the original repo is gone.

Sigmanificient commented 3 weeks ago

I think it should be dropped, as does not have any reverse dependency, and wasn't updated (release) since it as been moved to python modules 6 years ago.

emilazy commented 3 weeks ago

Seems like package weboob is a duplicate of woob?

Uhh, wow. The things you can find hiding in the corners of a repository this large. The one with a maintainer seems more polished and has a newer version and more current licence. But wow. We should definitely drop weboob; looks like upstream has used pytest since 2021, so it should be a pretty easy fix for the other one.

lavafroth commented 3 weeks ago

pylockfile is no longer maintained.

Sigmanificient commented 3 weeks ago

97 items: 45.36% done, 16.49% in progress, 37 waiting

moved down

emilazy commented 3 weeks ago

Maybe this will come in handy for stuff we need to patch: https://github.com/pytest-dev/nose2pytest.

I’ve packaged this in https://github.com/NixOS/nixpkgs/pull/330725 and am working on nose3 and its dependents.

pyrox0 commented 3 weeks ago

I'm working on python-hglib, which has needed me to update/modernize the python312Packages.glean-sdk and python312Packages.glean-parser derivations, and fix some mozphab tests. Should be submitting a PR as soon as nixpkgs-review finishes(again).

pyrox0 commented 3 weeks ago

python3Packages.docker doesn't depend on nose, and hasn't for a while, should be good to check off.(edit: was in the original list at the top of this issue, not the one lower down)

emilazy commented 3 weeks ago

Ah, that’s my script being bad. It’s docker-compose_1 in pkgs/applications/virtualization/docker/compose_1.nix. Which, uh, probably we could drop that?

Sigmanificient commented 3 weeks ago

115 items: 86.96% done, 11.30% in progress, 4 waiting

python3Packages

lavafroth commented 3 weeks ago

@emilazy python3Packages.mkl-service from the original list does not actually use nose. Guess we can get rid of it from the list.

emilazy commented 3 weeks ago

Looks like I just missed https://github.com/NixOS/nixpkgs/pull/324729. Fixed!

Sigmanificient commented 3 weeks ago

lockfile is no longer maintained.

Apache airflow depends on it tho: https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/apache-airflow/python-package.nix

emilazy commented 3 weeks ago

apache-airflow also has knownVulnerabilities set, so that’s fine.

Sigmanificient commented 3 weeks ago

What are its reverse dependencies?

emilazy commented 3 weeks ago

Nothing. But it doesn’t matter; Hydra won’t build it and users can’t use it without overriding a big scary warning anyway. If someone manages to get a newer, non‐vulnerable version working, it can be added back. Though of course they’d have to re‐add lockfile and fix its tests situation. I’m not necessarily opposed to just fixing lockfile but it’s unclear whether apache-airflow is going to be updated any time soon anyway. cc @risicle

Sigmanificient commented 3 weeks ago

Tried to make a patch for pydy:

diff --git a/pydy/codegen/tests/test_c_code.py b/pydy/codegen/tests/test_c_code.py
index 24b6784..5af9287 100644
--- a/pydy/codegen/tests/test_c_code.py
+++ b/pydy/codegen/tests/test_c_code.py
@@ -2,9 +2,10 @@

 import os

+import pytest
+
 from pkg_resources import parse_version
 import sympy as sm
-from nose.tools import assert_raises

 from ...models import multi_mass_spring_damper
 from ..c_code import CMatrixGenerator
@@ -40,7 +41,8 @@ class TestCMatrixGenerator():
         # Make sure an error is risen if not enough arguments are provided.
         arguments = self.arguments[:-1]

-        assert_raises(ValueError, CMatrixGenerator, arguments, self.matrices)
+        with pytest.raises(ValueError):
+            CMatrixGenerator(arguments, self.matrices)

     def test_generate_cse(self):

diff --git a/pydy/tests/test_utils.py b/pydy/tests/test_utils.py
index cc27c81..4b2bf9f 100644
--- a/pydy/tests/test_utils.py
+++ b/pydy/tests/test_utils.py
@@ -1,8 +1,9 @@
 #!/usr/bin/env python

+import pytest
+
 from pkg_resources import parse_version
 from setuptools import __version__ as SETUPTOOLS_VERSION
-from nose.tools import assert_raises

 from sympy import cos, sin, tan, sqrt, Matrix
 from sympy.physics.mechanics import dynamicsymbols
@@ -20,7 +21,7 @@ def test_sympy_equal_to_or_newer_than():
     assert sympy_equal_to_or_newer_than('0.6.5', '0.7.6.dev')
     assert not sympy_equal_to_or_newer_than('0.7.7', '0.7.6.dev')
     if parse_version(SETUPTOOLS_VERSION) >= parse_version('8.0'):
-        with assert_raises(ValueError):
+        with pytest.raises(ValueError):
             sympy_equal_to_or_newer_than('0.7.7', '0.7.6-git')

diff --git a/pydy/viz/tests/test_scene.py b/pydy/viz/tests/test_scene.py
index 9bb4c83..1673a64 100644
--- a/pydy/viz/tests/test_scene.py
+++ b/pydy/viz/tests/test_scene.py
@@ -4,11 +4,12 @@ import os
 import shutil
 import glob

+import pytest
+
 import numpy as np
 from numpy.testing import assert_allclose
 from sympy import symbols
 import sympy.physics.mechanics as me
-from nose.tools import assert_raises

 from ...system import System
 from ..shapes import Sphere
@@ -129,13 +130,13 @@ class TestScene(object):
         scene = Scene(self.ref_frame, self.origin, self.viz_frame,
                       times=self.sys.times)

-        with assert_raises(ValueError):
+        with pytest.raises(ValueError):
             scene.system = self.sys

         scene = Scene(self.ref_frame, self.origin, self.viz_frame,
                       system=self.sys)

-        with assert_raises(ValueError):
+        with pytest.raises(ValueError):
             scene.times = self.sys.times

     def test_clear_trajectories(self):
diff --git a/pydy/viz/tests/test_shapes.py b/pydy/viz/tests/test_shapes.py
index 767e0ce..487e061 100644
--- a/pydy/viz/tests/test_shapes.py
+++ b/pydy/viz/tests/test_shapes.py
@@ -2,9 +2,11 @@

 # external
 import numpy as np
+import pytest
+
 from sympy import symbols
 from numpy.testing import assert_allclose
-from nose.tools import assert_raises
+
 try:
     import pythreejs as p3js
 except ImportError:
@@ -29,19 +31,19 @@ def test_shape():

     shape.name = 'shape1'
     assert shape.name == 'shape1'
-    with assert_raises(TypeError):
+    with pytest.raises(TypeError):
         shape.name = 1

     shape.color = 'red'
     assert shape.color == 'red'
-    with assert_raises(ValueError):
+    with pytest.raises(ValueError):
         shape.color = 'puke'

     shape.material = "water"
     assert shape.material == "water"
     shape.material = "WATER"
     assert shape.material == "WATER"
-    with assert_raises(ValueError):
+    with pytest.raises(ValueError):
         shape.material = 'fluffy cloth'

     assert shape.generate_dict() == {"color": "red",
diff --git a/setup.py b/setup.py
index 0e230d6..5aeaacf 100644
--- a/setup.py
+++ b/setup.py
@@ -50,7 +50,7 @@ setup(
     packages=find_packages(),
     install_requires=install_requires,
     extras_require=extras_require,
-    tests_require=['nose>=1.3.7'],
+    tests_require=['pytest'],
     test_suite='nose.collector',
     include_package_data=True,
     classifiers=[

However there is many failling tests: 

=================================== test session starts ===================================
platform linux -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/sigmanificient/pydy
collected 71 items                                                                        

pydy/codegen/tests/test_c_code.py FFFFFFFFF                                         [ 12%]
pydy/codegen/tests/test_cython_code.py FFFF.                                        [ 19%]
pydy/codegen/tests/test_octave_code.py .                                            [ 21%]
pydy/codegen/tests/test_ode_function_generator.py ...FFFF..FFF...                   [ 42%]
pydy/tests/test_models.py ...                                                       [ 46%]
pydy/tests/test_system.py FFFFFFFFFFF.                                              [ 63%]
pydy/tests/test_utils.py ...                                                        [ 67%]
pydy/viz/tests/test_scene.py FFFFFFFF                                               [ 78%]
pydy/viz/tests/test_shapes.py ...............                                       [100%]

======================================== FAILURES =========================================
...
======================= 39 failed, 32 passed, 3 warnings in 35.88s ========================

https://github.com/pydy/pydy/issues/490

Sigmanificient commented 3 weeks ago

aws-google-auth may be dead: https://github.com/cevoaustralia/aws-google-auth/issues/284