Random failure in gaussian sampler

[vbraun]

Reported on sage-devel with sage-6.4.beta3:

sage -t --long src/sage/stats/distributions/discrete_gaussian_integer.pyx 
********************************************************************** 
File "src/sage/stats/distributions/discrete_gaussian_integer.pyx", 
line 262, in sage.stats.distributions.discrete_gaussian_integer.DiscreteGaussianDistributionIntegerSampler.__init__ 
Failed example: 
    min(l), max(l), abs(mean(l)-2.5) < 0.01 
Expected: 
    (-2, 7, True) 
Got: 
    (-2, 6, True) 

Specifically it happened once in make ptestlong. After that, 10/10 of sage -t --long src/sage/stats/distributions/discrete_gaussian_integer.pyx passed and 10/10 of manual input into Sage passed. The machine was an AMD Phenom X6 1055T with 8GB RAM running up-to-date OpenSuSE 12.3 (Linux 3.7.10-1.40-desktop).

This happened also at #15915 comment:54

CC: @malb @saraedum

Component: statistics

Keywords: random_fail

Author: Martin Albrecht

Branch/Commit: c566df1

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/16968

[vbraun]

Changed keywords from none to random_fail

[rwst]

Description changed:

--- 
+++ 
@@ -12,4 +12,6 @@
 Got: 
     (-2, 6, True) 

+Specifically it happened once in make ptestlong. After that, 10/10 of sage -t --long src/sage/stats/distributions/discrete_gaussian_integer.pyx passed and 10/10 of manual input into Sage passed. The machine was an AMD Phenom X6 1055T with 8GB RAM running up-to-date OpenSuSE 12.3 (Linux 3.7.10-1.40-desktop). + This happened also at #15915 comment:54

[malb]

Commit: cb9e4cd

[malb]

Branch: u/malb/t16968_random_gauss_failure

[malb]

comment:3

The doctest is too strict (given the non-deterministic behaviour), so I made it more lenient.

---

New commits:

cb9e4cd

More lenient doctest for non-deterministic gaussian sampler code

[vbraun]

comment:4

I guess the failure probability is a reasonably easy computation? Did you do it?

[kiwifb]

comment:5

While we are at follow up from that ticket. SAGE_ROOT doesn't belong to module_list.py anymore. If you need it you are doing something wrong.

--- module_list.py.orig 2014-09-12 15:03:19.370250091 +1200
+++ module_list.py  2014-09-12 15:02:13.930114901 +1200
@@ -1998,10 +1998,9 @@

     Extension('sage.stats.distributions.discrete_gaussian_integer',
               sources = ['sage/stats/distributions/discrete_gaussian_integer.pyx', 'sage/stats/distributions/dgs_gauss_mp.c', 'sage/stats/distributions/dgs_gauss_dp.c', 'sage/stats/distributions/dgs_bern.c'],
-              depends = ['sage/stats/distributions/dgs_gauss.h', 'sage/stats/distributions/dgs_bern.h'],
+              depends = ['sage/stats/distributions/dgs_misc.h', 'sage/stats/distributions/dgs_gauss.h', 'sage/stats/distributions/dgs_bern.h'],
               libraries = ['gmp', 'mpfr'],
               extra_compile_args=["-std=c99", "-D_XOPEN_SOURCE=600"],
-              include_dirs = [SAGE_ROOT +"/src/sage/stats/distributions"],
           ),

     ################################

I can get a different ticket, I am just venting my frustration a bit at spending my time removing them.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

a093d14

Better doctests of discrete Gaussians

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from cb9e4cd to a093d14

[malb]

comment:7

Replying to @kiwifb:

While we are at follow up from that ticket. SAGE_ROOT doesn't belong to module_list.py anymore. If you need it you are doing something wrong.

--- module_list.py.orig   2014-09-12 15:03:19.370250091 +1200
+++ module_list.py    2014-09-12 15:02:13.930114901 +1200
@@ -1998,10 +1998,9 @@

     Extension('sage.stats.distributions.discrete_gaussian_integer',
               sources = ['sage/stats/distributions/discrete_gaussian_integer.pyx', 'sage/stats/distributions/dgs_gauss_mp.c', 'sage/stats/distributions/dgs_gauss_dp.c', 'sage/stats/distributions/dgs_bern.c'],
-              depends = ['sage/stats/distributions/dgs_gauss.h', 'sage/stats/distributions/dgs_bern.h'],
+              depends = ['sage/stats/distributions/dgs_misc.h', 'sage/stats/distributions/dgs_gauss.h', 'sage/stats/distributions/dgs_bern.h'],
               libraries = ['gmp', 'mpfr'],
               extra_compile_args=["-std=c99", "-D_XOPEN_SOURCE=600"],
-              include_dirs = [SAGE_ROOT +"/src/sage/stats/distributions"],
           ),

     ################################

I can get a different ticket, I am just venting my frustration a bit at spending my time removing them.

Understood. This is now #16975

[vbraun]

comment:8

I also got the following from the buildbot right now:

sage -t --long src/sage/stats/distributions/discrete_gaussian_integer.pyx
**********************************************************************
File "src/sage/stats/distributions/discrete_gaussian_integer.pyx", line 262, in sage.stats.distributions.discrete_gaussian_integer.DiscreteGaussianDistributionIntegerSampler.__init__
Failed example:
    min(l), max(l), abs(mean(l)-2.5) < 0.01
Expected:
    (-2, 7, True)
Got:
    (-1, 7, True)
*********************************************************************

This is covered by the current branch, just putting it out for the record.

[vbraun]

comment:9

I got a new one:

sage -t --long --warn-long 61.7 src/sage/stats/distributions/discrete_gaussian_integer.pyx
**********************************************************************
File "src/sage/stats/distributions/discrete_gaussian_integer.pyx", line 256, in sage.stats.distributions.discrete_gaussian_integer.DiscreteGaussianDistributionIntegerSampler.__init__
Failed example:
    min(l)==2-2*1.0, max(l)==2+2*1.0, mean(l).n()
Expected:
    (True, True, 2.45...)
Got:
    (True, True, 2.46120071411133)
**********************************************************************

This is not fixed by the current branch...

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from a093d14 to c566df1

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

c566df1

Be more relaxed about testing discrete Gaussian 'dp' implementation

[malb]

comment:11

I committed a fix for this issue.

[vbraun]

Reviewer: Volker Braun

[vbraun]

Author: Martin Albrecht

[vbraun]

Changed branch from u/malb/t16968_random_gauss_failure to c566df1