Faster is_orthogonal_array

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

As the ticket claims this ticket implements a Cython version of is_orthogonal_array, which is then called by is_transversal_design and are_mutually_orthogonal_latin_squares.

Also sets a check=False in the code which slowed things down. As a resultthe tests are checked faster in the designs/ folder !

Nathann

Depends on #16236

CC: @videlec @KPanComputes @dimpase

Component: combinatorial designs

Author: Nathann Cohen

Branch/Commit: 02f1247

Reviewer: Vincent Delecroix, Brett Stevens

Issue created by migration from https://trac.sagemath.org/ticket/16295

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:46

I am just back from Scandinavia. I will do this today.

brett

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:47

I have made almost all the changes. In the code line

if verbose:
    print {"OA"   : "Columns {} and {} are not orthogonal".format(i,j),
           "MOLS" : "Matrices {} and {} are not orthogonal".format(i,j)}[terminology]
    return False

Is there a compact way of switching depending on the values of i and j in the case that terminology=MOLS? if i = 0 and j>1 then "square j-2" is not row latin. if i=1 and J> 1 then square j-2 is not column latin" if both i,j >2 then "square i-2 is not orthogonal to square j-2"

if i=0 and j=1 I have not found the right way to word the error message yet. Perhaps something like "the row and column indices are not orthogonal"

brett

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:48

Hello !

Replying to @brettpim:

I have made almost all the changes. In the code line

if verbose:
    print {"OA"   : "Columns {} and {} are not orthogonal".format(i,j),
           "MOLS" : "Matrices {} and {} are not orthogonal".format(i,j)}[terminology]
    return False

Is there a compact way of switching depending on the values of i and j in the case that terminology=MOLS? if i = 0 and j>1 then "square j-2" is not row latin. if i=1 and J> 1 then square j-2 is not column latin" if both i,j >2 then "square i-2 is not orthogonal to square j-2"

This block of code is not about each matrix being a latin square. Actually, this is not checked anywhere so I will add a commit.

This block of code is about the orthogonality of two matrices, i.e. do you find all possible pairs of integers when you read them simultaneously cell by cell.

Nathann

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:49

Nathann,

I understand what the code is doing. When I say "square" I mean "latin square" and in the language of MOLS column 0 of an OA is the row index of the Squares, column 1 is the column index and column i is latin square i-2. I am just making the error reporting consistent with the way that a MOLS person would interpret it. Please do not change the branch yet as I am in the middle of editing the code. I was just curious if there was a compact way to use a more complex case/switch in cython. But I think there is not so I will just make the changes with nested if statements.

brett

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:50

Still, it must be checked that each matrix is a latin square, because right now we have this :

sage: from sage.combinat.designs.latin_squares import are_mutually_orthogonal_latin_squares
sage: m1 = matrix([[0,0,0],[1,1,1],[2,2,2]])                                               
sage: m2 = matrix([[0,1,2],[0,1,2],[0,1,2]])                                               
sage: are_mutually_orthogonal_latin_squares([m1,m2], verbose=True)                         
True

I added a commit on u/ncohen/16295_v2 to fix that. Do not worry about possible conflicts with what you are doing : when you will be done, push your commit wherever you want on a branch of yours, give me its name and I will deal with the merge.

Nathann

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:51

Hello,

I understand what the code is doing. When I say "square" I mean "latin square" and in the language of MOLS column 0 of an OA is the row index of the Squares, column 1 is the column index and column i is latin square i-2.

Right now when are_mutually_latin_squares calls is_orthogonal_array it does not check that the resulting OA is a OA(k+2,n) but only that it is a OA(k,n). This, because the OA that is produced has only k columns and not k+2. This, because the columns corresponding to the row indices and the columns indices are not added to the OA.

This way, when the sequence of matrices is not a MOLS, the error message is that "matrix i is not orthogonal with matrix j" or that "matrix i is not a latin squares".

Which is, I believe, clearer to the user and easier to implement than by messing with the error messages of is_orthogonal_design.

Nathann

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:52

I just tried

$git trac push
git: 'trac' is not a git command. See 'git --help'.

but this did not work to push my revision up. What command do I use?

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:53

Hello !

but this did not work to push my revision up. What command do I use?

You seem to want to use "git trac" but it is not installed. This being said if you downloaded the branch without using "git trac" I don't think that just pushing is going to work.

What is the output of

git push trac HEAD:public/16295

It should push your current branch to public/16295.

Nathann

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:54

The Sage developers guide says "git trac puch" not "git push trac"

here is what I get when I do it the right way around:

$ git push trac
Counting objects: 13, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 937 bytes, done.
Total 7 (delta 6), reused 0 (delta 0)
remote: FATAL: W refs/heads/u/ncohen/16295 sage brett DENIED by fallthru
remote: error: hook declined to update refs/heads/u/ncohen/16295
To git@trac.sagemath.org:sage.git
 ! [remote rejected] u/ncohen/16295 -> u/ncohen/16295 (hook declined)
error: failed to push some refs to 'git@trac.sagemath.org:sage.git'

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:55

The Sage developers guide says "git trac puch" not "git push trac"

here is what I get when I do it the right way around:

Okay. So you noticed that the "right way" did not work. Can you try the "wrong way", i.e. the line I gave you above ? From the error message you got I think it will work.

The reason is simple : you cannot simply "push" to my branch because you have no write access to it. All branches whose name is u/my_login/whatever can only be written by guys whose login is my_login.

public/whatever branches can be written by anybody, though.

Nathann

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:56

your command worked but it creates a new branch and that is inoptimal

$ git push trac HEAD:public/16295
Counting objects: 13, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 937 bytes, done.
Total 7 (delta 6), reused 0 (delta 0)
To git@trac.sagemath.org:sage.git
 * [new branch]      HEAD -> public/16295

Does this get attached to the right ticket?

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:57

your command worked but it creates a new branch and that is inoptimal

Don't worry about branches, nobody has to pay for them. Anyway you cannot write on my branch (which is not my doing) so there is no other way.

Does this get attached to the right ticket?

It does not get attached to any ticket. You just added a branch on the trac server. Now I can look at it, we can discuss it, and if we agree with what is inside I will push it on the branch which is on this ticket. Or I will change this ticket's branch to public/16295 if it is easier.

Nathann

[dimpase]

comment:58

Replying to @brettpim:

your command worked but it creates a new branch and that is inoptimal

$ git push trac HEAD:public/16295
Counting objects: 13, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 937 bytes, done.
Total 7 (delta 6), reused 0 (delta 0)
To git@trac.sagemath.org:sage.git
 * [new branch]      HEAD -> public/16295

Does this get attached to the right ticket?

Why do you think this is not optimal? This is a normal way to go. Surely if there already was public/16295 branch you'd rather pushed there. But there was none. You can't push into someone else's branch. only into public/ ones.

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:59

Replying to @dimpase:

Why do you think this is not optimal? This is a normal way to go. Surely if there already was public/16295 branch you'd rather pushed there. But there was none. You can't push into someone else's branch. only into public/ ones.

I thought that that if there was a way for me to write to the patch's branch then having to create another is inoptimal, but if this is how it works then that is no problem.

[dimpase]

comment:60

Replying to @brettpim:

The Sage developers guide says "git trac puch" not "git push trac"

git trac means invoke git-trac git extension. (see http://github.com/sagemath/git-trac-command)

git push trac means do git's push to the remote namedtrac`.

(you can use git remote -v to see which remotes are set up for you)

here is what I get when I do it the right way around:

$ git push trac
Counting objects: 13, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 937 bytes, done.
Total 7 (delta 6), reused 0 (delta 0)
remote: FATAL: W refs/heads/u/ncohen/16295 sage brett DENIED by fallthru
remote: error: hook declined to update refs/heads/u/ncohen/16295
To git@trac.sagemath.org:sage.git
 ! [remote rejected] u/ncohen/16295 -> u/ncohen/16295 (hook declined)
error: failed to push some refs to 'git@trac.sagemath.org:sage.git'

surely, you can't push in an u/TRAC-USERNAME branch, that's why

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:61

Nathann,

I think if MOLS people are going to be using this code then we should add the additional 2 columns in when a MOLS gets transformed to a OA. In that case all the error messages I have changed are correct and this is what MOLS people will expect

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:62

Brett,

Here is the behaviour of your branch

sage: from sage.combinat.designs.latin_squares import are_mutually_orthogonal_latin_squares
sage: m1 = matrix([[0,1,0],[2,0,1],[1,2,0]]) # not a latin square
sage: m2 = matrix([[0,1,2],[1,2,0],[2,0,1]]) # not a latin square                                     
sage: are_mutually_orthogonal_latin_squares([m1,m2], verbose=True)                         
The row and column indices are not orthogonal
False
sage: m1 = matrix([[0,0,0],[1,1,1],[2,2,2]])  # not a latin square
sage: m2 = matrix([[0,1,2],[0,1,2],[0,1,2]])  # not a latin square
sage: are_mutually_orthogonal_latin_squares([m1,m2],verbose=True)
True
sage: a,b,c,d = designs.mutually_orthogonal_latin_squares(11,4)      
sage: are_mutually_orthogonal_latin_squares([a,b,c,d,d],verbose=True)
Squares 1 and 2 are not orthogonal
False

The index issues come from problem I mentionned in comment 51.

I think if MOLS people are going to be using this code then we should add the additional 2 columns in when a MOLS gets transformed to a OA. In that case all the error messages I have changed are correct and this is what MOLS people will expect

Here is what my branch u/ncohen/16295_v2 (mentionned in comment 50) does :

sage: from sage.combinat.designs.latin_squares import are_mutually_orthogonal_latin_squares
sage: m1 = matrix([[0,1,0],[2,0,1],[1,2,0]]) # not a latin square
sage: m2 = matrix([[0,1,2],[1,2,0],[2,0,1]]) # not a latin square
sage: are_mutually_orthogonal_latin_squares([m1,m2], verbose=True) 
Matrix 0 is not row latin
False
sage: m1 = matrix([[0,0,0],[1,1,1],[2,2,2]])  # not a latin square
sage: m2 = matrix([[0,1,2],[0,1,2],[0,1,2]])  # not a latin square
sage: are_mutually_orthogonal_latin_squares([m1,m2],verbose=True)
Matrix 0 is not row latin
False
sage: from sage.combinat.designs.latin_squares import are_mutually_orthogonal_latin_squares
sage: a,b,c,d = designs.mutually_orthogonal_latin_squares(11,4)
sage: are_mutually_orthogonal_latin_squares([a,b,c,d,d],verbose=True)
Matrices 3 and 4 are not orthogonal
False

Our terminology is almost the same (we can replace "matrix" with "square" if you prefer). The interest of mine is that you do not have to add MOLS-specific code to the is_orthogonal_array function. Thus do you mind if we keep my version of the code to make are_mutually_orthogonal_squares correct and include your typo fix ?

Nathann

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:63

Nathann,

I strongly feel that it would be much more consistent with

1) what people in design theory will expect the code to do

2) the equivalence between OAs and MOLS

to have are_mutually_orthogonal_squares add the two extra columns and then simply call is_orthogonal_array. This will have the effect of checking that all the squares are, in fact latin and you will not have to have are_mutually_orthogonal_squares do this extra checking.

in my opinion as a design theorist, the behaviour you cite in comment 62 of my code is all correct. In the second example the squares ARE orthogonal, they are just NOT latin. if you add the additional two columns the behaviour will be reported.

brett

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:64

Hello !

I strongly feel that it would be much more consistent with

1) what people in design theory will expect the code to do

You claim again that my code does something that somebody (a user ?) "in design theory" might not expect. Can you tell me what it is ? Or is the function okay for the user's point of view, while you think that the way it is coded is "something that people in design theory" would not expect ?

2) the equivalence between OAs and MOLS

to have are_mutually_orthogonal_squares add the two extra columns and then simply call is_orthogonal_array. This will have the effect of checking that all the squares are, in fact latin and you will not have to have are_mutually_orthogonal_squares do this extra checking.

I would have agreed if the function only returned True/False answer. What you are telling me to do is that I should consider all latins squares as k rows of a TD to which I add the two row/column ones, only to differentiate them later in a function that has no reason to contain MOLS-specific code.

This is why it does not make sense to me to build a homogeneous input only to dissassemble it immediately afterwards.

Besides, I expect that my commented code is sufficient for anybody "who does design theory" to understand what exactly happens.

in my opinion as a design theorist, the behaviour you cite in comment 62 of my code is all correct.

It is not. The function is named "are_mutually_orthogonal_latin_squares". It should return True if the input consists of mutually orthogonal latin squares.

In the second example the squares ARE orthogonal, they are just NOT latin. if you add the additional two columns the behaviour will be reported.

I agree with that, I am just showing that your branch returns wrong results. If this is fixed the results will be correct. This being said, if this is fixed then what you said just above, i.e. that the function should return True even when the input does not consist of latin squares, will not hold anymore. And so according to you the function will return wrong results.

Nathann

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:65

Two details that may help you decide :

1) The current behaviour of the are_mutually_orthogonal_latin_squares is to check that the latin squares are latin squares, and so changing that would mean changing the behaviour of the function

sage: from sage.combinat.designs.latin_squares import are_mutually_orthogonal_latin_squares
sage: are_mutually_orthogonal_latin_squares([Matrix([[1,1],[1,0]])])                       
False

2) All results returned by the constructors of mutually_orthogonal_latin_squares are checked using this function.. If it does not check whether the latin squares are orthogonal then we need to make sure that the output of our constructors are actually latin squares

Nathann

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:66

I updated my branch starting from u/ncohen/16295_v2 by adding your commit to it. I removed the "if" in is_orthogonal_array that only deal with MOLS input as this was already dealt with in are_mutually_orthogonal_latin_squares. This mixed version of our codes seems good to me...

Nathann

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 31e84b8 to b66a187

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

6fb1f14	trac #16295: Bugfix
1653790	trac #16295: Merged with 6.3.beta3
b66a187	fixed some small language typos and changed MOLS error messages into context that MOLS researchers will expect.

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:68

Brett ?

[videlec]

comment:69

Hi Nathann,

We waited too long for that ticket and many tickets are depending on this one. For me the review is basically done as the function does what is specified.

I added one nice doctest at u/vdelecroix/16295... If you like it, set my branch to positive review otherwise set yours to positive review.

Vincent

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

Changed commit from b66a187 to 02f1247

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:70

AHahahah. Funny doctest. I was actually thinking that "indeed, this function is used a LOT in the code, but that it is only tested on True instances. With your doctest, it makes it a bit more interesting than

def is_orthogonal_array(*args):
    return True

Thanks !

Nathann

---

New commits:

02f1247

trac #16295: one more doctest

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

Changed branch from u/ncohen/16295 to u/vdelecroix/16295

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

Reviewer: Vincent Delecroix, Brett Stevens

[65d8ac20-a56f-4d80-91b0-a0e64981c5c9]

comment:72

OK,

I am (mostly) happy. What do I do to approve this patch?

brett

[vbraun]

Changed branch from u/vdelecroix/16295 to 02f1247