doctest failure of sage_setup/clean.py

kiwifb commented 3 months ago

This is a good one.

sage -t --long --random-seed=7248854323037540409093786245466017113 /usr/lib/python3.12/site-packages/sage_setup/clean.py
**********************************************************************
File "/usr/lib/python3.12/site-packages/sage_setup/clean.py", line 92, in sage_setup.clean._find_stale_files
Failed example:
    for f in files('sagemath-standard'):
        dir = os.path.dirname(str(f))
        extra_files[dir] = extra_files.get(dir, [])
        extra_files[dir].append(str(f))
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.12/site-packages/sage/doctest/forker.py", line 715, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.12/site-packages/sage/doctest/forker.py", line 1147, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_setup.clean._find_stale_files[7]>", line 1, in <module>
        for f in files('sagemath-standard'):
    TypeError: 'NoneType' object is not iterable
**********************************************************************

Full relevant code

        sage: from importlib.metadata import files
        sage: for f in files('sagemath-standard'):
        ....:     dir = os.path.dirname(str(f))
        ....:     extra_files[dir] = extra_files.get(dir, [])
        ....:     extra_files[dir].append(str(f))

Documentation of importlib.metadata.files https://importlib-metadata.readthedocs.io/en/latest/api.html#importlib_metadata.Distribution.files

Discussion: so sage-on-gentoo installs a sagemath-standard-$version.dist-info directory and vanilla keeps the .egg file generated by sdist process. In the dist-info folder, sage-on-gentoo does not have a RECORD file (and vanilla has SOURCES.txt inside the .egg folder). The RECORD file is not compulsory but is usually produced by some of the python builders or the python installer. In our case setuptools does not produce a RECORD file but if you use pip to make a wheel, pip will add the appropriate RECORD file. When pip is used as a package manager, it needs RECORD to keep track of the installed files(you would think it is stored somewhere independently, but python provides for the existence of that file). Of course Linux distributions do not need this file because they track what is installed in some other ways. However, distributions that use pip to produce their packages will have a "RECORD" file as part of their installation.

What to do? It is unclear which part of the packaging is responsible for producing the metadata. Some other building backends will produce a RECORD file (example alabaster uses flit-core and does have a RECORD file). So, we cannot just go and say to gpep517 you need to finalise the build process by generating your own metadata because setuptools is not doing the full job. On the other hand, pip seems to do that as routine whatever the backend.

strogdon commented 2 months ago

How did you discover this? I don't see it when running the testsuite, nor when doctesting the file individually. It does appear when running the indicated code from the sage prompt, i.e. there are no files in sagemath-standard.

sage: from importlib.metadata import files
sage: files('sagemath-standard')
sage:

kiwifb commented 2 months ago

It is skipped in the latest beta because it has been marked as # needs SAGE_SRC at the top https://github.com/sagemath/sage/blob/744939e037a67193e730d7205e612e2d58197fca/src/sage_setup/clean.py#L83

cschwan / sage-on-gentoo

doctest failure of sage_setup/clean.py #782