search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.3k stars 447 forks source link

Install SAGE_SRC/ext/ in package data of sage package by setup.py, not SAGE_LOCAL/share/sage/ext/ by build/make/Makefile #21785

Closed mkoeppe closed 4 years ago

mkoeppe commented 7 years ago

A lot of the stuff (not necessarily all) in $SAGE_SRC/ext belongs directly in the sage Python package--in particular any files that are needed for the package itself to function properly.

A model for that is sage.gaprc, a file required for the libgap module to work, which is installed in sage.libs.gap as "package data" as of #22626.

In this ticket, we install all of ext as package data of sage.

Depends on #29233

CC: @jdemeyer @embray @kiwifb @slel @saraedum @isuruf @dimpase @jhpalmieri

Component: build

Author: Matthias Koeppe

Branch: fef5a5f

Reviewer: François Bissey

Issue created by migration from https://trac.sagemath.org/ticket/21785

kiwifb commented 7 years ago
comment:2

That would actually help me. I see extra potential to this for my jupyter kernel install woes as well (even if I know Jeroen doesn't see it that way).

slel commented 5 years ago
comment:3

There was a reference to this ticket in a "sage-packaging" discussion:

embray commented 5 years ago
comment:4

I would add, a lot of the stuff (not necessarily all) in $SAGE_SRC/ext might better belong directly in the sage Python package--in particular any files that are needed for the package itself to function properly. For example you can see in #22626 that I put sage.gaprc, a file required for the libgap module to work, in sage.libs.gap as "package data".

There are several files in $SAGE_SRC/ext that there's no reason to ever allow to be out of sync with the Python package, and that could be given the same treatment.

For files in $SAGE_SRC/ext in general, that aren't explicitly required from the package (are there any such files) it might be better to move out of src/ entirely?

embray commented 4 years ago
comment:5

Ticket retargeted after milestone closed

mkoeppe commented 4 years ago
comment:7

setuptools experts: To move this ticket forward, should it go through #22655 (Support package_data-like of non-Python resource files in Python packages)?

embray commented 4 years ago
comment:8

Something like it, yes, though I think maybe it can be simplified.

mkoeppe commented 4 years ago
comment:9

Erik, should discussion then take place on #22655 or here?

mkoeppe commented 4 years ago
comment:10

Description adapted from embray's comment 4.

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1 +1,9 @@
+A lot of the stuff (not necessarily all) in `$SAGE_SRC/ext` belongs directly in the sage Python package--in particular any files that are needed for the package itself to function properly. 

+A model for that is `sage.gaprc`, a file required for the `libgap` module to work, which is installed in `sage.libs.gap` as "package data" as of #22626.
+
+In this ticket, we install all of `ext` as package data of `sage`.
+
+- In a follow up ticket, if we identify any files in `$SAGE_SRC/ext` that aren't explicitly required from the package, we will move them to a more appropriate place.
+
+
mkoeppe commented 4 years ago

Dependencies: #29062

mkoeppe commented 4 years ago

Author: Matthias Koeppe

mkoeppe commented 4 years ago

Branch: u/mkoeppe/install_sage_src_extin_package_data_of_sage_package_by_setup_pynot_sage_local_share_sage_ext__by_build_make_makefile

mkoeppe commented 4 years ago
comment:13

Branch is on top of #29062.

Installation of the package data does not work yet... what am I missing?

New commits:

568afdcbuild/pkgs/python3/spkg-install: Install valgrind-python.supp directly in SAGE_LOCAL, not first in SAGE_SRC
ceb35bbsrc/bin/sage-valgrind: Find python.supp/python3.supp in SAGE_LOCAL/lib/valgrind or system directories
06a0226Move src/ext to src/sage/ext_data and install it as 'package_data'
mkoeppe commented 4 years ago

Commit: 06a0226

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

1edc654src/setup.py [package_data]: List all ext_data subdirectories
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 06a0226 to 1edc654

mkoeppe commented 4 years ago
comment:16

@embray Also this ticket would need review.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 1edc654 to 749928c

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

95c8316Move src/ext to src/sage/ext_data and install it as 'package_data'
749928csrc/setup.py [package_data]: List all ext_data subdirectories
mkoeppe commented 4 years ago
comment:18

Rebased on top of current #29062 (and 9.1.beta2)

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

278d68fMerge tag '9.1.beta3' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 749928c to 278d68f

mkoeppe commented 4 years ago
comment:20

Merged current beta, needs review

embray commented 4 years ago
comment:22

I haven't looked at this in detail yet, but I'm basically happy with what I see; this is what I've been wanting to see done all along.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 278d68f to 3995b6a

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

3995b6aMerge tag '9.1.beta4' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile
mkoeppe commented 4 years ago

Changed dependencies from #29062 to none

kiwifb commented 4 years ago
comment:25

Looks good but there are a few doctests errors on the patchbots and I think at least some of them are really related to this ticket

sage -t --long --warn-long 313.2 src/sage/ext_data/nbconvert/postprocess.py
    FileNotFoundError in doctesting framework
**********************************************************************
Traceback (most recent call last):
  File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2504, in __call__
    doctests, extras = self._run(runner, options, results)
  File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2549, in _run
    doctests, extras = self.source.create_doctests(sage_namespace)
  File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 734, in create_doctests
    load(filename, namespace) # errors raised here will be caught in DocTestTask
  File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/repl/load.py", line 252, in load
    exec(code, globals)
  File "./postprocess.py", line 19, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '-p'

Looking at this file, there is no tests in it, which suggests it may not have been tested before and may be nothing in ext_data should.

This one is actually the exact same error but on another file

sage -t --long --warn-long 313.2 src/sage/doctest/sources.py
**********************************************************************
File "src/sage/doctest/sources.py", line 765, in sage.doctest.sources.FileDocTestSource._test_enough_doctests
Failed example:
    for path, dirs, files in itertools.chain(os.walk('sage'), os.walk('doc')): # long time
        path = os.path.relpath(path)
        dirs.sort(); files.sort()
        for F in files:
            _, ext = os.path.splitext(F)
            if ext in ('.py', '.pyx', '.pxd', '.pxi', '.sage', '.spyx', '.rst'):
                filename = os.path.join(path, F)
                FDS = FileDocTestSource(filename, DocTestDefaults(long=True,optional=True))
                FDS._test_enough_doctests(verbose=False)
Exception raised:
    Traceback (most recent call last):
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.doctest.sources.FileDocTestSource._test_enough_doctests[6]>", line 9, in <module>
        FDS._test_enough_doctests(verbose=False)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 809, in _test_enough_doctests
        tests, _ = self.create_doctests({})
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 734, in create_doctests
        load(filename, namespace) # errors raised here will be caught in DocTestTask
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/repl/load.py", line 252, in load
        exec(code, globals)
      File "./postprocess.py", line 19, in <module>
    FileNotFoundError: [Errno 2] No such file or directory: '-p'

Something different

sage -t --long --warn-long 313.2 src/sage/homology/simplicial_set.py
**********************************************************************
File "src/sage/homology/simplicial_set.py", line 1679, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph
Failed example:
    CP3 = simplicial_sets.ComplexProjectiveSpace(3)
Exception raised:
    Traceback (most recent call last):
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.homology.simplicial_set.SimplicialSet_arbitrary.graph[6]>", line 1, in <module>
        CP3 = simplicial_sets.ComplexProjectiveSpace(Integer(3))
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 612, in ComplexProjectiveSpace
        data = simplicial_data_from_kenzo_output(file)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output
        with open(filename, 'r') as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/ext_data/kenzo/CP3.txt'
**********************************************************************
File "src/sage/homology/simplicial_set.py", line 1680, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph
Failed example:
    G = CP3.graph()
Exception raised:
    Traceback (most recent call last):
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.homology.simplicial_set.SimplicialSet_arbitrary.graph[7]>", line 1, in <module>
        G = CP3.graph()
    NameError: name 'CP3' is not defined
**********************************************************************
File "src/sage/homology/simplicial_set.py", line 1681, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph
Failed example:
    len(G.vertices())
Expected:
    1
Got:
    4
**********************************************************************
File "src/sage/homology/simplicial_set.py", line 1683, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph
Failed example:
    len(G.edges())
Expected:
    0
Got:
    6
**********************************************************************

Was a file lost in the move? And again in the same folder

sage -t --long --warn-long 313.2 src/sage/homology/simplicial_set_examples.py
**********************************************************************
File "src/sage/homology/simplicial_set_examples.py", line 535, in sage.homology.simplicial_set_examples.ComplexProjectiveSpace
Failed example:
    CP3 = simplicial_sets.ComplexProjectiveSpace(3)
Exception raised:
    Traceback (most recent call last):
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.homology.simplicial_set_examples.ComplexProjectiveSpace[1]>", line 1, in <module>
        CP3 = simplicial_sets.ComplexProjectiveSpace(Integer(3))
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 612, in ComplexProjectiveSpace
        data = simplicial_data_from_kenzo_output(file)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output
        with open(filename, 'r') as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/ext_data/kenzo/CP3.txt'
File "src/sage/homology/simplicial_set_examples.py", line 551, in sage.homology.simplicial_set_examples.ComplexProjectiveSpace
Failed example:
    simplicial_sets.ComplexProjectiveSpace(4).f_vector()
Exception raised:
    Traceback (most recent call last):
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.homology.simplicial_set_examples.ComplexProjectiveSpace[10]>", line 1, in <module>
        simplicial_sets.ComplexProjectiveSpace(Integer(4)).f_vector()
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 619, in ComplexProjectiveSpace
        data = simplicial_data_from_kenzo_output(file)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output
        with open(filename, 'r') as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/ext_data/kenzo/CP4.txt'
File "src/sage/homology/simplicial_set_examples.py", line 646, in sage.homology.simplicial_set_examples.simplicial_data_from_kenzo_output
Failed example:
    S4.homology(reduced=False)
Exception raised:
    Traceback (most recent call last):
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.homology.simplicial_set_examples.simplicial_data_from_kenzo_output[4]>", line 1, in <module>
        S4.homology(reduced=False)
    NameError: name 'S4' is not defined
kiwifb commented 4 years ago
comment:26

I have a felling the .txt in the kenso folder are not actually installed. At least they may not have been on the patchbot I have taken these logs from.

mkoeppe commented 4 years ago
comment:27

Yes, it looks like they are not getting installed. I'll investigate

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 3995b6a to ef3c7d3

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

ef3c7d3Fix install of ext_data/kenzo
mkoeppe commented 4 years ago
comment:29

Ok, the problem with kenzo was a simple typo.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from ef3c7d3 to 6967bc1

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

6967bc1Mark src/sage/ext_data to be skipped in doctesting
kiwifb commented 4 years ago
comment:31

I am trying to find if there was a reason it was named nodotest.py (specifically the .py part), this is an annoyance at install time, at least on sage-on-gentoo, for no good reason when examined closely.

kiwifb commented 4 years ago
comment:32

Sorry for the noise, I want to trigger the patchbot.

kiwifb commented 4 years ago
comment:34

Somewhat incidental, and doing it in another ticket would be fine. src/sage/doctest/tests is currently not installed because there is no __init__.py - at least on distros. But it contains a nodoctest.py which is tested for. So it should really be added to package_data as well.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 6967bc1 to 2c13f04

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

2c13f04src/setup.py: Add sage/doctest/tests to package_data
mkoeppe commented 4 years ago
comment:36

Next, to make it work, the cleaner of stale files needs to be tamed:

[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/__pycache__/nodoctest.cpython-37.pyc
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nodoctest.py
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/postprocess.py
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython-37.pyc
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/doctest/tests/__pycache__/nodoctest.cpython-37.pyc
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/doctest/tests/nodoctest.py
[sagelib-9.1.beta4] - cleaning build/lib.macosx-10.9-x86_64-3.7
[sagelib-9.1.beta4] Cleaning up stale file: build/lib.macosx-10.9-x86_64-3.7/sage/ext_data/nodoctest.py
[sagelib-9.1.beta4] Cleaning up stale file: build/lib.macosx-10.9-x86_64-3.7/sage/ext_data/nbconvert/postprocess.py
[sagelib-9.1.beta4] Cleaning up stale file: build/lib.macosx-10.9-x86_64-3.7/sage/doctest/tests/nodoctest.py
[sagelib-9.1.beta4] Finished cleaning, time: 0.25 seconds.
kiwifb commented 4 years ago
comment:37

OK I have somewhat asked on #14575 if there was any reason it was nodoctest.py - in particular the extension. I think we should have a separate ticket for this but the problem of __pycache__ with nodoctest would be eliminated. Still leave us with postprocess.py but at least, that one is genuine.

mkoeppe commented 4 years ago
comment:38

I'm working on the cleaner.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 2c13f04 to 86e2079

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

86e2079src/setup.py, src/sage_setup/clean.py: Clean carefully around package_data
mkoeppe commented 4 years ago
comment:40

Now it looks like this:

[sagelib-9.1.beta4] Cleaning up stale installed files....
[sagelib-9.1.beta4] - cleaning /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/__pycache__/nodoctest.cpython-37.pyc
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython-37.pyc
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/doctest/tests/__pycache__/nodoctest.cpython-37.pyc
[sagelib-9.1.beta4] - cleaning build/lib.macosx-10.9-x86_64-3.7
kiwifb commented 4 years ago
comment:41

New round of patchbot

sage -t --long --warn-long 286.7 src/sage/rings/function_field/function_field.py  # Timed out
sage -t --long --warn-long 286.7 src/sage/doctest/sources.py  # 1 doctest failed
sage -t --long --warn-long 286.7 src/sage_setup/clean.py  # 1 doctest failed

The time out is strange but was already there in the previous round so there may be something about it (but what?). The testing in doctest/sources.py is still present even so we don't test postprocess.py anymore so there is something more to do somewhere in there. And lastly we have some stale files - not unexpectedly at this stage

sage -t --long --warn-long 286.7 src/sage_setup/clean.py
**********************************************************************
File "src/sage_setup/clean.py", line 100, in sage_setup.clean._find_stale_files
Failed example:
    for f in stale_iter:
        if f.endswith(skip_extensions): continue
        print('Found stale file: ' + f)
Expected nothing
Got:
    Found stale file: sage/doctest/tests/nodoctest.py
    Found stale file: sage/ext_data/nodoctest.py
    Found stale file: sage/ext_data/nbconvert/postprocess.py
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 86e2079 to 50a5d25

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a22186fsrc/sage_setup/clean.py: Adjust _find_stale_files doctest
50a5d25Rename nodoctest.py to .nodoctest; respect both
mkoeppe commented 4 years ago
comment:43

This doctest is now fixed.

New commits:

a22186fsrc/sage_setup/clean.py: Adjust _find_stale_files doctest
50a5d25Rename nodoctest.py to .nodoctest; respect both