a bijectionist's toolkit

[mantepse]

We provide a toolkit for the combinatorialist to help find functions ("statistics") s: A -> Z and bijections A -> B given sequences of finite sets A and B that satisfy various constraints.

Depends on #34881 Depends on #34878

CC: @stumpc5 @tscrim @fchapoton

Component: combinatorics

Author: Alexander Grosz, Tobias Kietreiber, Stephan Pfannerer, Martin Rubey

Branch/Commit: u/mantepse/a_bijectionist_s_toolkit @ 53506aa

Reviewer: Matthias Koeppe, ...

Issue created by migration from https://trac.sagemath.org/ticket/33238

[mantepse]

Branch: u/mantepse/a_bijectionist_s_toolkit

[mantepse]

comment:4

Comments welcome! In particular, it would be wonderful to find more interesting applications.

In short: given two finite sets A and B, a statistic tau: B -> Z, and a set partition P of A, the program finds all possible statistics s: A -> Z which have the same distribution as tau but are constant on the blocks of P.

There is still some ongoing work. In particular, we try to implement a cache of solutions, which is checked for counterexamples before asking the solver to produce a counterexample from scratch.

---

New commits:

6d7ab9c

initial commit

[mantepse]

Commit: 6d7ab9c

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 6d7ab9c to 6149eb2

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

6149eb2

make the linter happier

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9d4bfb9

remove useless assignment

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 6149eb2 to 9d4bfb9

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

6ee34e1

make initialisation more efficient

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9d4bfb9 to 6ee34e1

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 6ee34e1 to 1a14bcf

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

1a14bcf

consistently name variable

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

0901ec7

slightly improve documentation

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1a14bcf to 0901ec7

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

c39d653

non-copying intersection, to save memory when there are almost no restrictions on the values of a block

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 0901ec7 to c39d653

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

2f0bb0e

Merge branch 'develop' of trac.sagemath.org:sage into t/33238/a_bijectionist_s_toolkit

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from c39d653 to 2f0bb0e

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

321ba43

start to cache solutions

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 2f0bb0e to 321ba43

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 321ba43 to 306395c

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

306395c

finish implementation of cache

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 306395c to 1ac0978

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

1a8e564	Merge branch 'develop' of trac.sagemath.org:sage into t/33238/a_bijectionist_s_toolkit
1ac0978	add some documentation and doctests, slightly simplify code

[mantepse]

comment:15

There are still some doctests and some documentation missing, but otherwise this is ready.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

47945ac

add missing documentation in table of contents

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1ac0978 to 47945ac

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 47945ac to 19c3d8f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

19c3d8f

mark doctests as long, slightly simplify logic

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

eca857e

slightly simplify, more doctests

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 19c3d8f to eca857e

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

a04d146

doctest _find_counter_example

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from eca857e to a04d146

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

3508426

doctest add_distribution_constraints and add_intertwing_relation_constraints

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from a04d146 to 3508426

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

0ac618c

doctest _preprocess_intertwining_relations, _solution, _show_bmilp, _initialize_new_bmilp, _veto_current_solution

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3508426 to 0ac618c

[mkoeppe]

comment:24

You may want to try this with a better IP solver - see #31329 for SCIP

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

d50b62c

expand docstring of main class

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 0ac618c to d50b62c

[mantepse]

comment:26

Replying to Matthias Köppe:

You may want to try this with a better IP solver - see #31329 for SCIP

Excellent, I have now managed to install SCIP, and indeed, many doctests fail :-)

Previously, I only tested with gurobi (which was much faster), but apparently, we got sloppy during the last few big reorganisations.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d50b62c to 288e391

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

288e391

copy milp instead of adding and removing constraints

[mantepse]

comment:29

With the simple fix from #31329, #31329 comment:45, this now also seems to work with SCIP. However, SCIP seems to be much slower, by far. In particular, GLPK finishes the doctest in line 3157-3171

    sage: As = Bs = [[],
    ....:            [(1,i,j) for i in [-1,0,1] for j in [-1,1]],
    ....:            [(2,i,j) for i in [-1,0,1] for j in [-1,1]],
    ....:            [(3,i,j) for i in [-2,-1,0,1,2] for j in [-1,1]]]

    sage: c1 = lambda a, b: (a[0]+b[0], a[1]*b[1], a[2]*b[2])
    sage: c2 = lambda a: (a[0], -a[1], a[2])

    sage: bij = Bijectionist(sum(As, []), sum(Bs, []))
    sage: bij.set_statistics((lambda x: x[0], lambda x: x[0]))
    sage: bij.set_intertwining_relations((2, c1, c1), (1, c2, c2))
    sage: l = list(bij.solutions_iterator()); len(l)                            # long time
    64

in about 1 minute on my laptop, while SCIP seems to take about an hour. Is there a way to find out what's going on? Even finding the first solution takes 25 seconds, vs. 0.3 seconds with GLPK.

[mantepse]

comment:30

I have found the culprit already - deepcopy takes forever:

sage: %prun next(bij.solutions_iterator())
         54830 function calls (54618 primitive calls) in 25.558 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1   24.908   24.908   24.908   24.908 {method '__deepcopy__' of 'sage.numerical.mip.MixedIntegerLinearProgram' objects}
   ...

vs.

sage: %prun next(bij.solutions_iterator())
         53403 function calls (53231 primitive calls) in 0.353 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.123    0.123    0.287    0.287 bijectionist.py:3005(add_intertwining_relation_constraints)
     4093    0.114    0.000    0.124    0.000 {method 'add_constraint' of 'sage.numerical.mip.MixedIntegerLinearProgram' objects}
        1    0.051    0.051    0.051    0.051 {method 'solve' of 'sage.numerical.mip.MixedIntegerLinearProgram' objects}
     4083    0.021    0.000    0.031    0.000 {built-in method builtins.sum}
     4372    0.010    0.000    0.010    0.000 <ipython-input-3-f7b4b09ab02e>:1(<lambda>)
    12008    0.009    0.000    0.009    0.000 bijectionist.py:3068(<genexpr>)
     4096    0.004    0.000    0.006    0.000 copy.py:66(copy)
     4082    0.004    0.000    0.004    0.000 {method 'is_zero' of 'sage.numerical.linear_functions.LinearFunction' objects}
        1    0.003    0.003    0.006    0.006 bijectionist.py:2026(_preprocess_intertwining_relations)
     4093    0.003    0.000    0.003    0.000 {built-in method builtins.isinstance}
        1    0.002    0.002    0.003    0.003 bijectionist.py:2892(add_alpha_beta_constraints)
     4093    0.002    0.000    0.004    0.000 linear_tensor.py:112(is_LinearTensor)
      194    0.001    0.000    0.001    0.000 bijectionist.py:2671(<genexpr>)
      628    0.001    0.000    0.001    0.000 {method 'find' of 'sage.sets.disjoint_set.DisjointSet_of_hashables' objects}
     4098    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
     4095    0.001    0.000    0.001    0.000 {method 'copy' of 'dict' objects}
        1    0.001    0.001    0.001    0.001 {method '__deepcopy__' of 'sage.numerical.mip.MixedIntegerLinearProgram' objects}

[mantepse]

comment:31

(I don't know whether copy would suffice for my purpose, but it takes forever, too)

Since this is almost certainly a SCIP problem, I'm going back to needs-review.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 288e391 to d57c8e5

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

d57c8e5

derandomize a test, mark example as random

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d57c8e5 to 7353fb5

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

7353fb5

correct typo, remove useless assignment

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 7353fb5 to db850f0