Speed up our test suite

[fingolfin]

In the CI, running our test suite takes between 15-25 minutes (Ubuntu, Julia 1.6, not including doctests). It would be nice to be faster, to reduce turnaround times for CI results. One option of course is to split the test suite into a "quick" and a "slow" version, and we may have to do that eventually. But in the meantime, perhaps we can speed up a few tests to make them faster. To this end, I had a look at our CI logs to get an idea which test groups are particularly "slow". Perhaps then people can look into some of them to determine if they could perhaps be made a bit faster, e.g. by testing smaller instances or fewer things. Of course we still want good test coverage, so it may very well be that the result is that we can't make it go faster, and that is fine. But we should at least look.

So, some "juicy bits", arbitrarily selecting test sets that take at least about 20 seconds, out of a CI test run that took 16:30 in total:

UPDATE: my original list was bad. Here is a new list, with the "worst" test sets in a recent test job, sorted by duration, as measured via "wallclock" (= timestamps provided in the GitHub Action log file)

• [ ] 01:41 Ring interface for localized polynomial rings (from test/Rings/mpoly-localizations.jl, CC @HechtiDerLachs)
• [ ] 01:26 Number field (from test/Rings/NumberField.jl, CC @thofma @fieker)
• [ ] 01:21 Polymake.factorisations (from test/NumberTheory/nmbthy-test.jl, CC @thofma, @fieker )
• [x] 01:21 Group - see PR #852 which brings it down to ~5 seconds
• [ ] 00:38 Polyhedron (from test/PolyhedralGeometry/Polyhedron.jl, CC @lkastner)
• [ ] 00:35 Direct products (from test/ToricVarieties/runtests.jl) (almost all of this is polymake compiling wrappers, see https://github.com/oscar-system/Oscar.jl/issues/851#issuecomment-986728726)
• [ ] 00:32 faithful reduction from char. zero to finite fields (from test/Groups/matrixgroups.jl)
• [ ] 00:27 Relationship group - forms
• [ ] 00:21 Binomial Ideals
• [ ] 00:19 mpoly-graded
• [ ] 00:19 Examples.galois_group
• [ ] 00:18 Primary invariants
• [ ] 00:18 Conjugation and centralizers

[thofma]

Might also be a good idea to run it locally using https://github.com/JuliaLang/julia/pull/43252

[tthsqe12]

So, have some global variable that the tests can read and determine how many tests to run?

[thofma]

So, have some global variable that the tests can read and determine how many tests to run?

Yes, this is one option. We could have a OSCAR_LONG_TEST environment variable, which sets test_long = true. Using this one can tweak the test runtime or hide a test completely behind a @test_long. We use this approach in Hecke. There is always one job doing the long tests.

[benlorenz]

10:24:38 Cone | 72 72 10:26:16 Test Summary: | Pass Total

10:26:16 Group | 9 9 10:26:58 Test Summary: | Pass Total

The extra empty lines are very confusing like this, the above means that Group took almost two minutes, but doesn't say anything about Cone. And whatever came after Group took about 42 seconds. Both Test Summary ... and <testname> | x x are printed at the same time and should be compared to the timestamp of the previous two such lines.

[thofma]

Here is the detailed runtime for every toplevel testset:

Test Summary: | Pass  Total  Duration
types         |  514    514    16.9 s
Test Summary: | Pass  Total  Duration
iterators     |  221    221    12.6 s
Test Summary: | Pass  Total  Duration
Cone          |   44     44     2.2 s
Test Summary: | Pass  Total  Duration
Group         |    9      9     2.7 s
Test Summary: | Pass  Total  Duration
Polyhedron    |   55     55     5.1 s
Test Summary: | Pass  Total  Duration
PolyhedralFan |   25     25     1.1 s
Test Summary:       | Pass  Total  Duration
SubdivisionOfPoints |   10     10     0.8 s
Test Summary: | Pass  Total  Duration
OscarPolytope |   52     52     4.7 s
Test Summary: | Pass  Total  Duration
Graphs        |   20     20     0.3 s
Test Summary: | Pass  Total  Duration
Serialization |   23     23     1.6 s
Test Summary: | Pass  Total  Duration
fmpz          |    7      7     0.3 s
Test Summary: | Pass  Total  Duration
fmpq          |   13     13     0.1 s
Test Summary: | Pass  Total  Duration
fmpz_mat      |    5      5     0.5 s
Test Summary: | Pass  Total  Duration
fmpq_mat      |    9      9     0.9 s
Test Summary:      | Pass  Total  Duration
single cyclotomics |    7      7     2.1 s
Test Summary:                    | Pass  Total  Duration
matrices over a cyclotomic field |   10     10     2.4 s
Test Summary: | Pass  Total  Duration
fmpz          |    4      4     0.1 s
Test Summary: | Pass  Total  Duration
fmpq          |    8      8     0.0 s
Test Summary: | Pass  Total  Duration
fmpz_mat      |    4      4     0.2 s
Test Summary: | Pass  Total  Duration
fmpq_mat      |    8      8     0.2 s
Test Summary:      | Pass  Total  Duration
single cyclotomics |    2      2     0.2 s
Test Summary:                    | Pass  Total  Duration
matrices over a cyclotomic field |    2      2     0.3 s
Test Summary:             | Pass  Total  Duration
GapGroup and GapGroupElem |    2      2     0.0 s
Test Summary:         | Pass  Total  Duration
Rings.ZZ.constructors |   14     14     0.1 s
Test Summary:       | Pass  Total  Duration
Rings.ZZ.properties |   41     41     0.1 s
Test Summary:       | Pass  Total  Duration
Rings.ZZ.arithmetic |   23     23     0.1 s
Test Summary:       | Pass  Total  Duration
Rings.ZZ.comparison |   50     50     0.1 s
Test Summary:     | Pass  Total  Duration
Rings.ZZ.divexact |   25     25     0.1 s
Test Summary:     | Pass  Total  Duration
Rings.ZZ.powering |   34     34     0.0 s
Test Summary:               | Pass  Total  Duration
Rings.ZZ.euclidean_division |   96     96     0.2 s
Test Summary:       | Pass  Total  Duration
Rings.ZZ.conversion |   17     17     0.0 s
Test Summary: | Pass  Total  Duration
Rings.ZZ.gcd  | 1205   1205     0.0 s
Test Summary:  | Pass  Total  Duration
Rings.ZZ.roots |   27     27     0.0 s
Test Summary:          | Pass  Total  Duration
Rings.ZZ.factorisation |   18     18     1.5 s
Test Summary:          | Pass  Total  Duration
Rings.ZZ.combinatorial |   91     91     0.1 s
Test Summary:               | Pass  Total  Duration
Rings.ZZ.number_theoretical |  103    103     0.1 s
Test Summary:         | Pass  Total  Duration
Rings.QQ.constructors |   68     68     0.1 s
Test Summary:       | Pass  Total  Duration
Rings.QQ.properties |   70     70     0.1 s
Test Summary:       | Pass  Total  Duration
Rings.QQ.arithmetic |   66     66     0.3 s
Test Summary:       | Pass  Total  Duration
Rings.QQ.comparison |   23     23     0.1 s
Test Summary:     | Pass  Total  Duration
Rings.QQ.divexact |   22     22     0.1 s
Test Summary:     | Pass  Total  Duration
Rings.QQ.powering |   23     23     0.0 s
Test Summary:               | Pass  Total  Duration
Polynomial ring constructor |   69     69     0.7 s
Test Summary:        | Pass  Total  Duration
Polynomial Orderings |   18     18     2.9 s
Test Summary:   | Pass  Total  Duration
Polynomial homs |    4      4    10.4 s
Test Summary:    | Pass  Total  Duration
Ideal operations |   10     10     1.0 s
Test Summary:         | Pass  Total  Duration
Primary decomposition |   32     32     6.1 s
Test Summary: | Pass  Total  Duration
Groebner      |    7      7    13.2 s
Test Summary:         | Pass  Total  Duration
Primary decomposition |   11     11     0.1 s
Test Summary: | Pass  Total  Duration
#795          |    2      2     0.0 s
Test Summary:         | Pass  Total  Duration
algebra homomorphisms |   11     11     5.1 s
Test Summary:                | Pass  Total  Duration
finiteness tests for alghoms |    3      3     0.5 s
Test Summary:         | Pass  Total  Duration
subalgebra membership |    2      2     0.2 s
Test Summary: | Pass  Total  Duration
#655          |    1      1     0.0 s
Test Summary:   | Pass  Total  Duration
MPolyQuo.graded |    1      1     1.1 s
Test Summary: | Pass  Total  Duration
mpoly-graded  | 1203   1203    22.0 s
Test Summary: | Pass  Total  Duration
Coercion      |    2      2     0.0 s
Test Summary: | Pass  Total  Duration
Evaluation    |    1      1     0.3 s
Test Summary: | Pass  Total  Duration
Promotion     |    1      1     0.0 s
Test Summary: | Pass  Total  Duration
Degree        |    1      1     0.0 s
Test Summary: | Pass  Total  Duration
Grading       |    1      1     1.2 s
Test Summary:          | Pass  Total  Duration
mpoly-loc constructors |    2      2     0.8 s
Test Summary:        | Pass  Total  Duration
mpoly-loc operations |   10     10     3.2 s
Test Summary:      | Pass  Total  Duration
mpoly-loc groebner |    2      2     0.0 s
Test Summary:       | Pass  Total  Duration
mpoly-localizations |   24     24     2.8 s
Test Summary:                        | Pass  Total  Duration
mpoly-localizations PowersOfElements |   12     12     1.1 s
Test Summary:                    | Pass  Total  Duration
mpoly-localization homomorphisms |    5      5     1.6 s
Test Summary:                                 |  Pass  Total  Duration
Ring interface for localized polynomial rings | 17277  17277     1.1 m
Test Summary:         | Pass  Total  Duration
integer-localizations |   30     30     1.2 s
Test Summary:      | Pass  Total  Duration
nmod-localizations |   11     11     0.3 s
Test Summary:         | Pass  Total  Duration
mpoly-nested.frac.gcd |   10     10     3.1 s
Test Summary:            | Pass  Total  Duration
mpoly-nested.frac.factor |   10     10     4.2 s
Test Summary:                    | Pass  Total  Duration
mpoly-nested.iterated.conversion |    2      2     0.7 s
Test Summary:                    | Pass  Total  Duration
mpoly-nested.iterated.gcd_factor |   30     30     7.6 s
Test Summary: | Pass  Total  Duration
MPolyQuo      |    6      6     0.5 s
Test Summary:         | Pass  Total  Duration
MpolyQuo.manipulation |    7      7     2.8 s
Test Summary:   | Pass  Total  Duration
MPolyQuo.ideals |   19     19     0.4 s
Test Summary: | Pass  Total  Duration
msolve        |   13     13     3.5 s
Test Summary:   | Pass  Total  Duration
FractionalIdeal |   10     10     0.2 s
Test Summary:                       | Pass  Total  Duration
mpoly_affine_algebras.normalization |   56     56     2.3 s
Test Summary:                        | Pass  Total  Duration
mpoly_affine_algebras.integral_basis |   12     12     1.4 s
Test Summary: | Pass  Total  Duration
LazyPolyRing  |    3      3     0.0 s
Test Summary: | Pass  Total  Duration
LazyPoly      |   16     16     0.0 s
Test Summary: | Pass  Total  Duration
SLPolyRing    |   64     64     2.8 s
Test Summary: | Pass  Total  Duration
SLPoly        |  414    414     8.4 s
Test Summary:           | Pass  Total  Duration
Polymake.factorisations |    2      2     1.4 m
Test Summary:          | Pass  Total  Duration
Polymake.norm_equation |    1      1     2.1 s
Test Summary:                   | Pass  Total        Duration
GAPGroups_interface_conformance |   67     67  2.73077345e7 m
Test Summary:                   | Pass  Total        Duration
GAPGroups_interface_conformance |   67     67  2.73077345e7 m
Test Summary:                   | Pass  Total        Duration
GAPGroups_interface_conformance |   67     67  2.73077346e7 m
Test Summary:                   | Pass  Total        Duration
GAPGroups_interface_conformance |   67     67  2.73077346e7 m
Test Summary:                   | Pass  Total        Duration
GAPGroups_interface_conformance |   66     66  2.73077346e7 m
Test Summary:                   | Pass  Total        Duration
GAPGroups_interface_conformance |   66     66  2.73077346e7 m
Test Summary: | Pass  Total  Duration
Iteration     |   21     21     0.1 s
Test Summary:                | Pass  Total  Duration
The groups Sym(n) and Alt(n) |  120    120     0.3 s
Test Summary:        | Pass  Total  Duration
Special Constructors |   31     31     0.6 s
Test Summary:    | Pass  Total  Duration
Classical groups |   99     99     3.1 s
Test Summary:           | Pass  Total  Duration
Operations on PermGroup |  288    288     0.0 s
Test Summary:       | Pass  Total  Duration
Matrix manipulation |   23     23     3.1 s
Test Summary:                 | Pass  Total  Duration
Operations with vector spaces |   13     13     3.8 s
Test Summary:   | Pass  Total  Duration
Stuff on fields |    9      9     1.4 s
Test Summary: | Pass  Total  Duration
Permutations  |  150    150     0.2 s
Test Summary:    | Pass  Total  Duration
Change of parent |   27     27     0.1 s
Test Summary: | Pass  Total  Duration
Eltypes       |   26     26     1.1 s
Test Summary: | Pass  Total  Duration
Generators    |   22     22     0.2 s
Test Summary:             | Pass  Total  Duration
Finitely presented groups |   21     21     0.6 s
Test Summary: | Pass  Total  Duration
Subgroups     |   45     45     1.4 s
Test Summary:                          | Pass  Total  Duration
Centralizers and Normalizers in Sym(n) |   44     44     0.8 s
Test Summary: | Pass  Total  Duration
Cosets        |   73     73     1.8 s
Test Summary:         | Pass  Total  Duration
Predicates for groups |   19     19     0.5 s
Test Summary:            | Pass  Total  Duration
Sylow and Hall subgroups |   53     53     0.8 s
Test Summary:           | Pass  Total  Duration
Some specific subgroups |   10     10     0.6 s
Test Summary:                         | Pass  Total  Duration
Conjugacy classes in symmetric groups |  132    132     1.8 s
Test Summary:                | Pass  Total  Duration
Conjugation and centralizers |  725    725    19.1 s
Test Summary:                              | Pass  Total  Duration
Conjugation and centralizers for GL and SL |   63     63     2.7 s
Test Summary:          | Pass  Total  Duration
Homomorphism in Sym(6) |   23     23     0.4 s
Test Summary:                        | Pass  Total  Duration
Operations on homomorphism in Sym(6) |   29     29     0.1 s
Test Summary:     | Pass  Total  Duration
Isomorphic groups |  292    292     0.3 s
Test Summary:  | Pass  Total  Duration
Direct product |   75     75     2.3 s
Test Summary:       | Pass  Total  Duration
Kernel and cokernel |   58     58     1.7 s
Test Summary:                | Pass  Total  Duration
Automorphism group of Sym(n) |   33     33     1.2 s
Test Summary:              | Pass  Total  Duration
Other automorphisms groups |    2      2     0.2 s
Test Summary: | Pass  Total  Duration
Transitivity  |   22     22     1.4 s
Test Summary:  | Pass  Total  Duration
Perfect groups |    8      8     0.4 s
Test Summary: | Pass  Total  Duration
Small groups  |   15     15     0.4 s
Test Summary:  | Pass  Total  Duration
Directproducts |   81     81     2.4 s
Test Summary:      | Pass  Total  Duration
Semidirectproducts |   22     22     0.8 s
Test Summary:  | Pass  Total  Duration
Wreathproducts |   31     31     1.1 s
Test Summary:                            |   Pass   Total  Duration
Oscar-GAP relationship for finite fields | 133341  133341     4.9 s
Test Summary:                                | Pass  Total  Duration
Oscar-GAP relationship for cyclotomic fields | 1783   1783     3.8 s
Test Summary:                                       | Pass  Total  Duration
faithful reduction from char. zero to finite fields |   64     64     6.1 s
Test Summary:   | Pass  Total  Duration
Type operations |    4      4     0.0 s
Test Summary:     | Pass  Total  Duration
Fields assignment |   45     45     0.5 s
Test Summary: | Pass  Total  Duration
Constructors  |  247    247     2.7 s
Test Summary:              | Pass  Total  Duration
Assignments and generators |   25     25     0.1 s
Test Summary: | Pass  Total  Duration
Iterator      |    2      2     0.2 s
Test Summary: | Pass  Total  Duration
Membership    |   28     28     0.7 s
Test Summary:       | Pass  Total  Duration
Methods on elements |   24     24     0.1 s
Test Summary: | Pass  Total  Duration
Subgroups     |   12     12     0.3 s
Test Summary:                | Pass  Total  Duration
Cosets and conjugacy classes |   19     19     1.3 s
Test Summary:    | Pass  Total  Duration
Jordan structure |   85     85     2.0 s
Test Summary:                | Pass  Total  Duration
G-sets of permutation groups |   68     68     3.1 s
Test Summary:                        | Pass  Total  Duration
natural action of permutation groups |   28     28     0.4 s
Test Summary:    | Pass  Total  Duration
Definition forms |   45     45     1.8 s
Test Summary:    | Pass  Total  Duration
Evaluating forms |   68     68     0.1 s
Test Summary:      | Pass  Total  Duration
Methods with forms |   13     13     0.6 s
Test Summary: | Pass  Total  Duration
TransformForm |   38     38     3.4 s
Quadratic form with Gram matrix
Quadratic form with Gram matrix
Quadratic form with Gram matrix
Quadratic form with Gram matrix
Symmetric form with Gram matrix
Symmetric form with Gram matrix
Symmetric form with Gram matrix
Hermitian form with Gram matrix
Test Summary:              | Pass  Total  Duration
Relationship group - forms |  109    109    12.6 s
Test Summary:                          | Pass  Total  Duration
labelled_matrix_formatted, text format |   18     18     2.8 s
Test Summary:                           | Pass  Total  Duration
labelled_matrix_formatted, LaTeX format |   14     14     0.6 s
Test Summary:                   | Pass  Total  Duration
show and print character tables |   12     12     5.5 s
Test Summary:           | Pass  Total  Duration
create character tables |    3      3     0.1 s
Test Summary: | Pass  Total  Duration
characters    |   33     33     1.8 s
Test Summary: | Pass  Total  Duration
Number field  | 3709   3709     1.5 m
Test Summary: | Pass  Total  Duration
FunctionField |   60     60     5.5 s
Test Summary:  | Pass  Total  Duration
AbelianClousre |  655    655     3.8 s
Test Summary:   | Pass  Total  Duration
Binomial Ideals |   32     32    21.7 s
Test Summary:         | Pass  Total  Duration
Examples.galois_group |    3      3    17.7 s
Test Summary:     |Duration
Examples.ModStdQt | No tests   1.3 s
Test Summary:     | Pass  Total  Duration
Examples.ModStdNF |    3      3     1.9 s
Test Summary:         | Pass  Total  Duration
Modules: Constructors |    1      1     0.3 s
Test Summary:           | Pass  Total  Duration
Intersection of modules |    8      8     7.0 s
Test Summary: | Pass  Total  Duration
Presentation  |   50     50     6.6 s
Test Summary: | Pass  Total  Duration
Test kernel   |   19     19     2.1 s
Test Summary:  | Pass  Total  Duration
iszero(SubQuo) |    7      7     0.1 s
Test Summary:        | Pass  Total  Duration
simplify subquotient |   56     56     1.9 s
Test Summary:    | Pass  Total  Duration
quotient modules |   24     24     1.4 s
Test Summary: | Pass  Total  Duration
submodules    |   18     18     0.4 s
Test Summary: | Pass  Total  Duration
Hom module    |  150    150     4.6 s
Test Summary:       | Pass  Total  Duration
tensoring morphisms |   40     40     6.0 s
Test Summary:  | Pass  Total  Duration
direct product |   42     42     4.2 s
Test Summary:      | Pass  Total  Duration
Coordinates (lift) |   64     64     4.8 s
Test Summary:        | Pass  Total  Duration
module homomorphisms |   37     37    10.6 s
Test Summary: | Pass  Total  Duration
preimage      |    5      5     1.1 s
Test Summary:  | Pass  Total  Duration
InvariantRings |   42     42    10.9 s
Test Summary:      | Pass  Total  Duration
Primary invariants |    9      9    18.5 s
Test Summary:          | Pass  Total  Duration
Affine toric varieties |   14     14     1.4 s
Test Summary:                                                    | Pass  Total  Duration
Affine toric varieties created as general normal toric varieties |    2      2     0.0 s
Test Summary:                 | Pass  Total  Duration
Cyclic quotient singularities |    5      5     0.1 s
Test Summary:                        | Pass  Total  Duration
Toric varieties from polyhedral fans |    4      4     2.0 s
Test Summary:    | Pass  Total  Duration
Projective space |   16     16     2.9 s
Test Summary:      | Pass  Total  Duration
Hirzebruch surface |   30     30     5.3 s
Test Summary:     | Pass  Total  Duration
delPezzo surfaces |    8      8     0.7 s
Test Summary:              | Pass  Total  Duration
Blowup of projective space |   15     15     0.2 s
Test Summary:   | Pass  Total  Duration
Direct products |   17     17     0.3 s
Test Summary:                 | Pass  Total  Duration
ComparisonWithProjectiveSpace |    6      6     0.6 s
Test Summary: | Pass  Total  Duration
Divisors      |   21     21     0.4 s
Test Summary:         | Pass  Total  Duration
Polytopes of divisors |    2      2     0.0 s

[fingolfin]

My apologies, I indeed misinterpreted the test outputs (whose presentation IMHO isn't that great on multiple levels... but I digress sigh). I've updated my original post with a hopefully better and easier to read list. Sorry for the confusion

[benlorenz]

The output is really confusing yes.

Test Summary: | Pass Total Duration Group | 9 9 2.7 s Test Summary: | Pass Total Duration Polyhedron | 55 55 5.1 s

I am quite confused by these times being way different than the other times, this is without #852 ?

[fingolfin]

The timings reported by @thofma (AFAICT they were produced before #852) are indeed weird, and don't match what I see in CI and locally judging by wall clock time. Of course my machine has a different speed etc.; and I think those timings exclude compilation time (I think it's useful to separate that, but we still would like to know about it). But still: e.g. Direct products is listed with 0.3 s, but in my measurements it's more like 35 s, which is two orders of magnitude different. Can that really be explained by compilation time?!

[HechtiDerLachs]

About the localization interface: My 'own' tests should take only seconds. What really eats up the time is calling the 'ring interface test' due to @tthsqe12 . I think it was planned to include a switch there and provide a shorter test set for that? If yes, I'll be happy to switch that to 'short'. Otherwise, we could simply take out the test for the ring interface for now.

[thofma]

The timings I posted were obtained using https://github.com/JuliaLang/julia/pull/43252 and Oscar 0.7. I believe that they do not include compilation time. I don't understand how Direct products takes more than 1 a second in your table. I get (in a fresh session and Oscar master branch):

julia> P2 = toric_projective_space(2)
HA normal toric variety corresponding to a polyhedral fan in ambient dimension 2

julia> H5 = hirzebruch_surface(5)
A normal toric variety corresponding to a polyhedral fan in ambient dimension 2

julia> v = H5 * P2
A normal toric variety corresponding to a polyhedral fan in ambient dimension 4

julia> @time @testset "Direct produts" begin
           @test isnormal(v) == true
           @test isaffine(v) == false
           @test isprojective(v) == true
           @test issmooth(v) == true
           @test iscomplete(v) == true
           @test hastorusfactor(v) == false
           @test isorbifold(v) == true
           @test issimplicial(v) == true
           @test ith_betti_number(v, 0) == 1
           @test ith_betti_number(v, 1) == 0
           @test ith_betti_number(v, 2) == 3
           @test ith_betti_number(v, 3) == 0
           @test ith_betti_number(v, 4) == 4
           @test ith_betti_number(v, 5) == 0
           @test ith_betti_number(v, 6) == 3
           @test ith_betti_number(v, 7) == 0
           @test ith_betti_number(v, 8) == 1
         end
Test Summary: | Pass  Total
blub          |   17     17
  0.768306 seconds (914.85 k allocations: 49.043 MiB, 21.60% compilation time)
Test.DefaultTestSet("Direct products", Any[], 17, false, false)

Or am I looking at the wrong test?

EDIT: If I change the test file and put a @time in front of this testset, I get (during the ]test Oscar run)

Direct products |   17     17
  0.332782 seconds (72.88 k allocations: 638.500 KiB)

EDIT 2: See https://github.com/oscar-system/Oscar.jl/issues/851#issuecomment-986728726 for an explanation.

[lkastner]

So is the PolyhedralGeometry part done and the PR is just written in the wrong line or did it really fix Group?

[benlorenz]

Latest nightly test runs will show timings by default: https://github.com/oscar-system/Oscar.jl/runs/4424684401?check_suite_focus=true#step:5:139

[thofma]

@benlorenz Could you do me a favor and try to reproduce the timings from Direct products (from test/ToricVarieties/runtests.jl)? With recent Oscar master and julia 1.6 I saw 20 seconds once, but then only < 1 seconds (always in fresh julia sessions so always including compilation time). Since julia does not cache something itself (except for the precompiled stuff), is there something cached by polymake somewhere that could explain these numbers?

[benlorenz]

It might be because of wrappers the polymake needs to compile, these will be stored below .julia/polymake_user (and very soon in a scratchspace). You could run rm -rf ~/.julia/polymake_user/wrappers.* and rerun the tests.

[thofma]

Thanks, I just checked and it is as you suspected. This explains the discrepancy between the timings I reported and the original ones by @fingolfin, at least for things related to polymake.

[benlorenz]

Thanks for confirming. We should keep at least one test-case that triggers one wrapper compilation to make sure that still works, as we previously had some cases where this failed within Oscar but worked in Polymake.jl alone. But I think we currently have two (Direct products + Polyhedron) so we might adjust one testcase to not trigger a new wrapper (or pregenerate it in polymake_jll).

(Unfortunately compiling a wrapper is quite slow on github CI, > 20 sec)

[benlorenz]

As discussed in the Friday-meeting the runtime of the Ring interface for localized polynomial rings does jump up quite a bit from time to time.

I let this group run for a while and got about 500 iterations:

• 300 are below 1 minute
• 130 between 1 and 2 minutes
• 55 between 2 and 5 minutes
• 8 between 5 and 10 minutes
• 5 runs with more than 10 minutes

The longest one I have seen so far is 20m28.2s on a rather fast machine. The full testsuite runs in about 16 minutes (on average), so maybe twice as fast as the CI. So these 20 minutes could translate to about 40 minutes in the CI, and many more in pkg-eval / nanosoldier.

The long runs:

Ring interface for localized polynomial rings | 18768  18768  15m42.4s
Ring interface for localized polynomial rings | 18726  18726  20m28.2s
Ring interface for localized polynomial rings | 18816  18816  11m43.1s
Ring interface for localized polynomial rings | 18768  18768  11m44.9s
Ring interface for localized polynomial rings | 18864  18864  12m13.0s

While I was running this the Ring conformance tests failed once but I will create a separate ticket for this.

And I will also try to post an updated list of the longest test-groups soon, once I have parsed and analyzed the output...

[benlorenz]

Some more timings (top 20):

Number of iterations: 200
Sorted by median, max shown if max >= median*1.05, in seconds:
                                Polyhedron{nf_elem}: median  70.2
      Ring interface for localized polynomial rings: median  53.9 - max  1465.1
                                       Number field: median  44.3
                             PolyhedralFan{nf_elem}: median  42.3
                              MPolyAnyMap/MPolyRing: median  33.9 - max    35.9
                                     AbelianClousre: median  23.6 - max    24.8
                                   Polyhedron{fmpq}: median  21.9
                         PolyhedralComplex{nf_elem}: median  17.9
                                        Polynomials: median  15.7 - max    16.7
                               Experimental.gmodule: median  15.1
                  Direct product of toric varieties: median  14.8
                                             Timing: median  13.3
                                           Matroids: median  11.9
                                       Presentation: median  11.6
     Orthogonal groups of torsion quadratic modules: median  11.3
                          Experimental.galois_group: median  10.0
                                       mpoly-graded: median   9.9
                       Conjugation and centralizers: median   9.2
                                    Binomial Ideals: median   8.6 - max     9.1
                         Relationship group - forms: median   7.2 - max     7.7

Unfortunately julia only shows the name of the testgroup which makes it sometimes difficult to find where they belong. Polyhedron{nf_elem}, PolyhedralFan{nf_elem}, PolyhedralComplex{nf_elem}, and probably Polyhedron{fmpq} should become a lot faster with the the upcoming polymake 4.7 release.

[fingolfin]

Thanks for the analysis, @benlorenz ! Perhaps we should adjust all our @testset to reference the filename in which they occur as part of their name? We could even wrote a variant of the @testset macro for doing that...

Regarding the problem with "Ring interface for localized polynomial rings" (CC @HechtiDerLachs), my guess is that these computations involve Gröbner basis computations, so depending on the input the speed can vary dramatically. This could e.g be addressed by not using random inputs there but rather specifying a list of inputs to use. I hope @HechtiDerLachs can take a look at that.

[HechtiDerLachs]

I can look into that, yes. But from what I remember, the ring interface asks you to implement some "random-elements-generator" for the ring and through certain parameters, you can roughly regulate the degree of the polynomials involved. I'm afraid, these are no good means to reasonably bound the complexity of the tests, since, as you say, Groebner basis computations can behave quite wild.

Anyway: I can try to play a bit with the parameters and see what I get.

[benlorenz]

Why can't we seed these random-elements such that the testsuite will run on exactly the same random elements every time? (Or at least add in an environment variable that forces this / sets the seed in the CI)

[thofma]

The ring interface function is basically testing only arithmetic (and maybe polynomials and matrices, but this is also only ring arithmetic). I am not sure, but does that really do a Gröbner basis computation every time? I would have guessed that it does at most a few such computation and its complexity depends on the data defining the localization and not on the elements for which one does arithmetic. Maybe the ring itself is created using random parameters?

[fingolfin]

I think just deciding equality for these elements requires a Gröbner basis computation.

[HechtiDerLachs]

I was thinking about it again (sorry, I did not yet have the time to really look into it), but the localizations of polynomial rings should in fact not require any Groebner basis computations, really. At least, I tried to do everything using divides and ppio and the like. But the root is the generation of random elements and for this I was using the generation of random multivariate polynomials. There were three ranges that I had to specify as inputs, but I don't recall that there was a seed that I could specify. If such a feature exists, I think, I can use it.