Distributions sagemath-{flint-arb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules

mkoeppe commented 4 years ago

Basically, we are making all external libraries optional, except for a minimal set of "core" libraries such as cysignals and gmpy2 (see #29865).

(from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sagemath-ntl) - but some libraries come in packs that would not make much sense to separate (such as sagemath-flint-arb-e_antic).

List of distributions:

sagemath-flint-arb
- 'sage/libs/arb/arb_version.pyx',
- 'sage/libs/arb/arith.pyx',
- 'sage/matrix/matrix_complex_ball_dense.pyx',
- 'sage/rings/complex_arb.pyx',
- 'sage/rings/number_field/number_field_element_quadratic.pyx',
- 'sage/rings/polynomial/polynomial_complex_arb.pyx',
- 'sage/rings/real_arb.pyx'
- etc.
sagemath-homfly: 'sage/libs/homfly.pyx'
sagemath-giac (ex giacpy_sage in sage.libs.giac + dependency on libgiac)
- sage/libs/giac/giac.pyx
sagemath-gap:
- 'sage/coding/codecan/codecan.pyx',
- 'sage/combinat/enumeration_mod_permgroup.pyx',
- 'sage/combinat/root_system/reflection_group_c.pyx',
- 'sage/combinat/root_system/reflection_group_element.pyx',
- 'sage/graphs/spanning_tree.pyx',
- 'sage/groups/libgap_wrapper.pyx',
- 'sage/groups/matrix_gps/group_element.pyx',
- 'sage/groups/perm_gps/permgroup_element.pyx',
- 'sage/groups/perm_gps/partn_ref/automorphism_group_canonical_label.pyx',
- 'sage/groups/perm_gps/partn_ref/canonical_augmentation.pyx',
- 'sage/groups/perm_gps/partn_ref/data_structures.pyx',
- 'sage/groups/perm_gps/partn_ref/double_coset.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_binary.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_graphs.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_lists.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_matrices.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_python.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_sets.pyx',
- 'sage/groups/perm_gps/partn_ref2/refinement_generic.pyx',
- 'sage/libs/gap/element.pyx',
- 'sage/libs/gap/libgap.pyx',
- 'sage/libs/gap/util.pyx',
- 'sage/matrix/matrix_gap.pyx',
- 'sage/sets/disjoint_set.pyx'

To test:

make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_flint_arb
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_gap
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_giac
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_homfly

In follow-up tickets:

sagemath-brial (ex sage-brial in sage.rings.polynomial.pbori + sage.libs.polybori, sage.crypto.boolean_function)
sagemath-ntl
sagemath-ecl (sage.libs.ecl)
sagemath-maxima: requires sage-ecl
sagemath-singular: (some of these files actually may be better put in the pynac distribution)
- 'sage/algebras/letterplace/free_algebra_element_letterplace.pyx',
- 'sage/algebras/letterplace/free_algebra_letterplace.pyx',
- 'sage/algebras/letterplace/letterplace_ideal.pyx',
- 'sage/libs/pynac/constant.pyx',
- 'sage/libs/pynac/pynac.pyx',
- 'sage/libs/singular/function.pyx',
- 'sage/libs/singular/groebner_strategy.pyx',
- 'sage/libs/singular/option.pyx',
- 'sage/libs/singular/polynomial.pyx',
- 'sage/libs/singular/ring.pyx',
- 'sage/libs/singular/singular.pyx',
- 'sage/matrix/matrix_mpolynomial_dense.pyx',
- 'sage/rings/polynomial/multi_polynomial_ideal_libsingular.pyx',
- 'sage/rings/polynomial/multi_polynomial_libsingular.pyx',
- 'sage/rings/polynomial/plural.pyx',
- 'sage/symbolic/comparison.pyx',
- 'sage/symbolic/constants_c.pyx',
- 'sage/symbolic/expression.pyx',
- 'sage/symbolic/function.pyx',
- 'sage/symbolic/getitem.pyx',
- 'sage/symbolic/ring.pyx',
- 'sage/symbolic/series.pyx',
- 'sage/symbolic/substitution_map.pyx',
sagemath-pynac
sagemath-linbox:
- 'sage/libs/linbox/linbox_flint_interface.pyx',
- 'sage/matrix/matrix_integer_sparse.pyx',
- 'sage/matrix/matrix_modn_dense_double.pyx',
- 'sage/matrix/matrix_modn_dense_float.pyx',
- 'sage/matrix/matrix_modn_sparse.pyx'
sagemath-znpoly (subject to removal in #32841?)
- 'sage/modular/modsym/p1list.pyx',
- 'sage/modular/pollack_stevens/dist.pyx',
- 'sage/rings/fraction_field_FpT.pyx',
- 'sage/rings/polynomial/polynomial_zmod_flint.pyx',
- 'sage/schemes/hyperelliptic_curves/hypellfrob.pyx',
sagemath-mari (also requiring pbori and pbori-groebner):
- 'sage/matrix/matrix_gf2e_dense.pyx',
- 'sage/matrix/matrix_integer_dense.pyx',
- 'sage/matrix/matrix_mod2_dense.pyx',
- 'sage/matrix/matrix_rational_dense.pyx',
- 'sage/modules/vector_mod2_dense.pyx',
- 'sage/rings/polynomial/pbori/pbori.pyx',
- 'sage/rings/polynomial/polynomial_gf2x.pyx'

This is preparation for Meta-ticket #29705 (Modularization) and #30371.

Depends on #31031 Depends on #34855 Depends on #34839

CC: @tobiasdiez @dimpase

Component: refactoring

Keywords: sd111

Author: Matthias Koeppe

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30666

mkoeppe commented 1 year ago

Author: Matthias Koeppe

dimpase commented 1 year ago

comment:42

How are all these tags like # sage_setup: distribution = sagemath-gap processed?

mkoeppe commented 1 year ago

comment:43

They are processed by the functions in sage_setup.find, via sage.misc.package_dir.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1. New commits:

133e5b8 Remove copy-pasted conda.txt files

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from 0f198ca to 133e5b8

mkoeppe commented 1 year ago

comment:45

sage -t --random-seed=146157697718740975867546676674797881386 sage/groups/perm_gps/partn_ref/data_structures.pyx
**********************************************************************
File "sage/groups/perm_gps/partn_ref/data_structures.pyx", line 1221, in sage.groups.perm_gps.partn_ref.data_structures.SC_test_list_perms
Failed example:
    for i in range(2,9):
        test_Sn_on_m_points(i,i,1,0)
Expected nothing
Got:
    ImportError: cannot import name n_is_prime
    Exception ignored in: 'sage.groups.perm_gps.partn_ref.data_structures.SC_is_giant'
    Traceback (most recent call last):
      File "<doctest sage.groups.perm_gps.partn_ref.data_structures.SC_test_list_perms[2]>", line 4, in test_Sn_on_m_points

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from 133e5b8 to 4c8f5b5

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1. New commits:

4c8f5b5 src/sage/sets/disjoint_set.pyx: Remove obsolete -*- coding -*- line

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1. New commits:

2be77af pkgs/sagemath-flint-arb: Update description

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from 4c8f5b5 to 2be77af

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1. New commits:

a827f29 pkgs/sagemath-giac: Update description

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from 2be77af to a827f29

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from a827f29 to 112e29e

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1. New commits:

112e29e src/sage/groups/perm_gps/partn_ref/data_structures.pyx: In doctest support code, avoid use of FLINT

mkoeppe commented 1 year ago

Description changed:

--- 
+++ 
@@ -44,8 +44,10 @@

 To test:

+- `make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_flint_arb`
 - `make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_gap`
-
+- `make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_giac`
+- `make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_homfly`

 In follow-up tickets:

dimpase commented 1 year ago

comment:53

there are a number of files with m4 suffix, which does not seem to be m4 files. Should they rather get in suffux?

mkoeppe commented 1 year ago

comment:54

They are processed by m4, see build/pkgs/sagelib/bootstrap

dimpase commented 1 year ago

comment:55

Imho they should have .in suffix. Otherwise, looks OK.

mkoeppe commented 1 year ago

comment:56

The .in files in use with autotools are not processed by m4.

These .m4 files are processed by m4. For example, in setup.cfg.m4 you'll see the m4 macro call esyscmd.

dimpase commented 1 year ago

comment:57

ah, OK. I misunderstood what "processed" means in this context.

mkoeppe commented 1 year ago

comment:58

Thanks!

mkoeppe commented 1 year ago

Reviewer: Dima Pasechnik

kiwifb commented 1 year ago

comment:60

When I build the full sagelib with this ticket included (but I do not build those sub distribution yet) in sage-on-gentoo, building the documentation fails. After inspection, I have the sources installed for sage/libs/flint, sage/libs/arb etc, but none of compiled module are installed. Is there something I am missing or doing wrong?

mkoeppe commented 1 year ago

comment:61

Are you building from src/ or from pkgs/sagemath-standard/?

kiwifb commented 1 year ago

comment:62

src with the setup.py from sagemath-standard. This is when I build against develop or the Volker merging branch. The main reason I am starting from src in those cases, is that patch does not like links.

kiwifb commented 1 year ago

comment:63

I also delete src/sage_setup so it is not used instead of the installed one.

mkoeppe commented 1 year ago

comment:64

OK, that's something that I'll have to fix in pkgs/sagemath-standard/setup.py. Thanks for catching this

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1. New commits:

`790c12b`	`Merge tag '9.8.beta5' into t/30666/add__sage_setup__distribution__directives_to_all_cython_modules_needing_external_libraries`
`8b1cf25`	`pkgs/sagemath-standard/setup.py: Include files marked as # sage_setup: distribution = sagemath-flint-arb etc.`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from 112e29e to 8b1cf25

mkoeppe commented 1 year ago

Changed dependencies from #31031 to #31031, #34855

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from 8b1cf25 to 1531fda

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`2b39b3d`	`Merge #34858`
`c59622a`	`build/pkgs/sagemath*/spkg-src: Use build`
`7fc93d0`	`build/pkgs/sagemath*/spkg-src: Use --skip-dependency-check`
`4940db4`	`build/make/Makefile.in: Add targets SPKG-src for script packages`
`573fcfd`	`build/make/Makefile.in: Move targets pypi-sdists, wheel, pypi-wheels here; use SPKG-sdist targets`
`88e18d0`	`Merge #34855`
`0bd13ac`	`build/make/Makefile.in: Remove duplicate setting of PYPI_WHEEL_PACKAGES`
`f712d67`	`Merge #34855`
`5fc1add`	`build/pkgs/sagemath*/spkg-src: Use build`
`1531fda`	`build/pkgs/sagemath_flint_arb/dependencies: Fix up`

mkoeppe commented 1 year ago

Changed dependencies from #31031, #34855 to #31031, #34855, #34839

mkoeppe commented 1 year ago

Removed branch from issue description; replaced by PR #35663

sagemath / sage

Distributions sagemath-{flint-arb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules #30666