setuptools >=65 + cython = warnings

[kiwifb]

Lots of doctest failures are appearing now that setuptools 65+ are the only version availables. cython called distutils extension that are deprecated. Example

sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/rings/polynomial/ore_polynomial_element.pyx
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/rings/polynomial/ore_polynomial_element.pyx", line 1310, in sage.rings.polynomial.ore_polynomial_element.OrePolynomial._left_lcm_cofactor
Failed example:
    cython('''  # optional - sage.misc.cython
    from sage.rings.polynomial.ore_polynomial_element cimport OrePolynomial
    def left_lcm_cofactor(OrePolynomial P, OrePolynomial Q):
        return P._left_lcm_cofactor(Q)
    ''')
Expected nothing
Got:
    doctest:warning
      File "/usr/lib/python-exec/python3.10/sage-runtests", line 154, in <module>
        err = DC.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 1379, in run
        self.run_doctests()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 1054, in run_doctests
        self.dispatcher.dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2021, in dispatch
        self.parallel_dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1916, in parallel_dispatch
        w.start()  # This might take some time
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2190, in start
        super().start()
      File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 281, in _Popen
        return Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71, in _launch
        code = process_obj._bootstrap(parent_sentinel=child_r)
      File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
        self.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2162, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2492, in __call__
        doctests, extras = self._run(runner, options, results)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2544, in _run
        result = runner.run(test)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 866, in run
        return self._run(test, compileflags, out)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.polynomial.ore_polynomial_element.OrePolynomial._left_lcm_cofactor[0]>", line 1, in <module>
        cython('''  # optional - sage.misc.cython
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 673, in cython_compile
        return cython_import_all(tmpfile, get_globals(), **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 563, in cython_import_all
        m = cython_import(filename, **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 534, in cython_import
        name, build_dir = cython(filename, **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 295, in cython
        from Cython.Build import cythonize
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/Cython/Build/__init__.py", line 1, in <module>
        from .Dependencies import cythonize
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 41, in <module>
        import pythran
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/pythran/__init__.py", line 41, in <module>
        from pythran.config import get_include
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/pythran/config.py", line 8, in <module>
        import numpy.distutils.system_info as numpy_sys
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/numpy/distutils/system_info.py", line 196, in <module>
        from numpy.distutils.command.config import config as cmd_config
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/numpy/distutils/command/config.py", line 19, in <module>
        from numpy.distutils.mingw32ccompiler import generate_manifest
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/numpy/distutils/mingw32ccompiler.py", line 28, in <module>
        from distutils.msvccompiler import get_build_version as get_build_msvc_version
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/setuptools/_distutils/msvccompiler.py", line 66, in <module>
        warnings.warn(
      File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
        sw(msg.message, msg.category, msg.filename, msg.lineno,
    :
    DeprecationWarning: msvccompiler is deprecated and slated to be removed in the future. Please discontinue use or file an issue with pypa/distutils describing your use case.
**********************************************************************

Harmless but pollute the test results each time cython is called.

[kiwifb]

All those are most likely cython calls

sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/sagedoc.py  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/graphs/connectivity.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/cachefunc.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/sageinspect.py  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/repl/ipython_extension.py  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/env.py  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/superseded.py  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/graphs/graph_decompositions/fast_digraph.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/rings/tate_algebra_ideal.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/arith/long.pxd  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/rings/polynomial/ore_polynomial_element.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/cpython/getattr.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/rings/integer_fake.pxd  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/structure/factory.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/features/__init__.py  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/symbolic/pynac.pxi  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/cpython/cython_metaclass.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/cpython/wrapperdescr.pxd  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/inherit_comparison.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/lazy_attribute.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/docs/instancedoc.py  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/session.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/cpython/string.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/instancedoc.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/misc/nested_class.pyx  # 1 doctest failed
sage -t --long --random-seed=205705670410541188758081033436036560950 /usr/lib/python3.10/site-packages/sage/structure/element.pyx  # 1 doctest failed

[strogdon]

Also, from the sage prompt

sage: cython('''
....: ''')
/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/setuptools/_distutils/msvccompiler.py:66: DeprecationWarning: msvccompiler is deprecated and slated to be removed in the future. Please discontinue use or file an issue with pypa/distutils describing your use case.
  warnings.warn(

[strogdon]

And is not present when run a second time.

[kiwifb]

That normal, warnings are only supposed to appear once in a session. I think.

[strogdon]

Unless I'm mistaken, vanilla restricts setuptools to <64.0 (see https://trac.sagemath.org/ticket/34442), but for other reasons.

[kiwifb]

Strange reasons because I build numpy with setuptools 65 just fine. I am guessing it doesn't work with the options they want.

[strogdon]

It is curious that these setuptools related failures to cython calls do not fail when run individually, e.g.

sage -t --long /usr/lib/python3.10/site-packages/sage/graphs/connectivity.pyx
too many failed tests, not using stored timings
Running doctests with ID 2022-10-19-23-48-57-ed854ed1.
Running with SAGE_LOCAL='/usr' and SAGE_VENV='/usr'
Using --optional=sage
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_cubic_hecke,database_jones_numfield,database_knotinfo,dvipng,gfan,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,msolve,nauty,palp,pandoc,pdf2svg,pdftocairo,phitigra,plantri,polytopes_db,polytopes_db_4d,pynormaliz,python_igraph,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.misc.cython,sage.plot,sage.rings.number_field,sage.rings.padics,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,sphinx,tdlib
Doctesting 1 file.
/usr/lib/python3.10/site-packages/setuptools/_distutils/msvccompiler.py:66: DeprecationWarning: msvccompiler is deprecated and slated to be removed in the future. Please discontinue use or file an issue with pypa/distutils describing your use case.
  warnings.warn(
sage -t --long --random-seed=338177061023648646321341562693635303826 /usr/lib/python3.10/site-packages/sage/graphs/connectivity.pyx
    [511 tests, 8.88 s]
----------------------------------------------------------------------
All tests passed!

[kiwifb]

There may actually be i/o flushing issues. The warning is partly printed.

[strogdon]

Is this setuptools-related or something else? I no longer see the warnings for the original issue.

sage -t --long --random-seed=291285384843659035697685462386874152650 /usr/lib/python3.11/site-packages/sage/repl/ipython_kernel/kernel.py
**********************************************************************
File "/usr/lib/python3.11/site-packages/sage/repl/ipython_kernel/kernel.py", line 47, in sage.repl.ipython_kernel.kernel.SageKernel.__init__
Failed example:
    from sage.repl.ipython_kernel.kernel import SageKernel
Expected nothing
Got:    
    0.00s - Debugger warning: It seems that frozen modules are being used, which may
    0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
    0.00s - to python to disable frozen modules.
    0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
...

From the sage prompt:

sage: from sage.repl.ipython_kernel.kernel import SageKernel
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
/usr/lib/python3.11/site-packages/pydevd_plugins/extensions/__init__.py:4: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydevd_plugins.extensions')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  __import__('pkg_resources').declare_namespace(__name__)
/usr/lib/python3.11/site-packages/pydevd_plugins/extensions/types/__init__.py:4: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydevd_plugins.extensions.types')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  __import__('pkg_resources').declare_namespace(__name__)

No failure on vanilla where an older setuptools is used.

[strogdon]

I have no idea where the emoji came from.

[kiwifb]

The emoji seems to be default in github these days. The stuff you point out to is from pydebug. Some recent changes have made it quieter I think. A lot of things have gone quieter, someone did something about it.

[strogdon]

I thought perhaps pydebug but no vanilla failure which was odd.

[kiwifb]

May be it was not pydebug in isolation. But that was definitely pydebug messages.

[strogdon]

I believe the DeprecationWarning has been fixed here https://github.com/microsoft/debugpy/commit/00bc5136e4697b86cf99bcfbb1a1a9a2c44eabe0 and the fix is probably in debugpy-1.6.7.

[strogdon]

debugpy-1.6.7 doesn't appear to help.

[kiwifb]

After inspection, I still have that message. It is just buried under other warnings from pydevd, which may the real python debugging package at work.

[strogdon]

If I delete line 2338:

warnings.warn(msg, DeprecationWarning, stacklevel=2)

in /usr/lib/python3.11/site-packages/pkg_resources/__init__.py I see no DeprecationWarning

$ sage -t --long /usr/lib/python3.11/site-packages/sage/repl/ipython_kernel/kernel.py 
too many failed tests, not using stored timings
Running doctests with ID 2023-04-21-13-02-21-bbbdc6e9.
Running with SAGE_LOCAL='/usr' and SAGE_VENV='/usr'
Using --optional=sage
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,cvxopt,cvxopt,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_cubic_hecke,database_jones_numfield,database_knotinfo,dvipng,fpylll,gfan,graphviz,imagemagick,ipython,jupymake,kenzo,latte_int,lrcalc_python,lrslib,mcqd,meataxe,mpmath,msolve,nauty,networkx,numpy,palp,pandoc,pdf2svg,pdftocairo,pexpect,phitigra,pillow,plantri,polytopes_db,polytopes_db_4d,pplpy,primecountpy,ptyprocess,pynormaliz,python_igraph,requests,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.gap,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modules,sage.plot,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.real_double,sage.rings.real_mpfr,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,scipy,singular,sphinx,sympy,tdlib
Doctesting 1 file.
sage -t --long --random-seed=268197945637086208363461818001488521182 /usr/lib/python3.11/site-packages/sage/repl/ipython_kernel/kernel.py
    [12 tests, 0.25 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 0.3 seconds
    cpu time: 0.2 seconds
    cumulative wall time: 0.2 seconds
Features detected for doctesting: 

The __init__.py file is provided by setuptools-67.6.1. I believe the DeprecationWarning originates with something to do with implicit namespace packages (PEP 420) which I don't fully understand.

[strogdon]

The setup.py patch at https://github.com/microsoft/debugpy/pull/1244 fixes here the debugpy/pydevd related doctest failures for

sage -t --long /usr/lib/python3.11/site-packages/sage/repl/ipython_kernel/kernel.py

when using >=setuptools -67.3.0. The above debugpy patch also includes changes to the _vendored folder. But gentoo ultimately removes this folder.

[kiwifb]

That's interesting. The file is the last one with failures in my last round of testing.

[strogdon]

I have networkx-3.1 failures in sage/graphs/graph.py. This should be resolved with https://github.com/sagemath/sage/pull/35584. And I have the Timed out failure from issue https://github.com/cschwan/sage-on-gentoo/issues/707.

[strogdon]

Does https://github.com/sagemath/sage/pull/35831 fix the pkg_resources DeprecationWarnings?

[strogdon]

I still get the DeprecationWarning.

[kiwifb]

Sorry, I have been a bit busy (and sneezy). They are gone for me for some reasons. Which one are you still getting?

[strogdon]

DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydevd_plugins.extensions.types')`.
    Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages

when doctesting

sage -t --long /usr/lib/python3.11/site-packages/sage/repl/ipython_kernel/kernel.py

The patch at https://github.com/cschwan/sage-on-gentoo/issues/717#issuecomment-1561900297 does remove the warning but probably not the best thing to do.

[kiwifb]

OK I still have something there too. It has to be that.

[kiwifb]

I do not think the patch you mention is the right way to go indeed. The package at fault is really pydevd. It is the one that raises warnings about pkg_resources. And it does not look like upstream is doing anything about it - or at least properly. The patch may delay the debugger attachment which gets rid of all those pesky messages.

[strogdon]

The frozen chatter can be removed by inserting

    warnings.filterwarnings("ignore", category=DeprecationWarning)

as it appears in /usr/lib/python3.11/site-packages/pydevd_plugins/__init__.py in

/usr/lib/python3.11/site-packages/pydevd_plugins/extensions/__init__.py
/usr/lib/python3.11/site-packages/pydevd_plugins/extensions/types/__init__.py

I'm not sure why the warnings.filterwarnings should be missing from these latter two __init__.py files.

[kiwifb]

Yes, it is not very consistent.

[kiwifb]

Fixed now