Test consistency between R and python implementations

[pneuvial]

(this can be done from the R or python side)

[alexblnn]

I have worked on a notebook to do this, but it requires consistent seeding between python and rpy2. For now we don't know how to do this. Perhaps this could be done the other way around using reticulate

[pneuvial]

I believe this is not possible the other way around either: the random number generators are different.

One workaround would be to pass the matrix of label permutations all_shuffled_labels to R. It is not output by the function get_permuted_p_values` but you could:

1. run the calibration via python (with proper rng seeding)

2. generate the shuffled labels outside the function by running (after the same seeding) as in https://github.com/pneuvial/sanssouci.python/blob/45062bcbd21a3796954061f4def8c3b32fd3b526/sanssouci/lambda_calibration.py#L60-L63

3. pass this matrix to R (via rpy2) and run rowWelchTests, whose second argument can be a matrix of shuffled labels, e.g.:

perm <- rowWelchTests(X, null_groups)
calib <- calibrate(perm$p.value, 0.1, family = "Linear")

Note that this should also work with rowZTests too, for one-sample tests.

[pneuvial]

TODO:

• check consistency between get_pivotal_statistics (no RNG involved)
• check consistency between get_permuted_p_values (one and two-sample) on a small example where the permutations are quasi deterministic