tests hang on Windows

[breisfeld]

Enthought Python Distribution -- www.enthought.com Version: 7.1-1 (32-bit)

Python 2.7.2 |EPD 7.1-1 (32-bit)| (default, Jul 3 2011, 15:13:59) [MSC v.1500 32 bit (Intel)] on win32 Type "packages", "demo" or "enthought" for more information.

import statsmodels statsmodels.version '0.4.0.dev-96d5979'

C:: nosetests statsmodels C:\Python27\lib\site-packages\statsmodels-0.4.0-py2.7-win32.egg\statsmodels\tools\tools.py:300: FutureWarning: The default of prepend will be change d to True in 0.5.0, use explicit prepend FutureWarning) C:\Python27\lib\site-packages\statsmodels-0.4.0-py2.7-win32.egg\statsmodels\tsa\vector_ar\var_model.py:340: FutureWarning: The names argument is depre cated and will be removed in the next release. "removed in the next release.", FutureWarning) ...............................................................................................S...................................................... .......................................S............................................................................F................................. ...................................................................................................................................................... ...................................................................................................................................................... .............................................................................................S........S..........S........S.........S..........S...... ..S.........S.........S............................................................................................................................... ...............................................................................................F...................................................

The tests just hang at this point and never complete.

Platform: Windows 7

[josef-pkt]

which scipy version scipy.__version__

and please run the tests with verbose

nosetests -v statsmodels

so we can see which test hangs.

there were some problems with endless loops in scipy.optimize.fmin_bfgs on older versions of scipy but I thought we skip them.

[breisfeld]

Enthought Python Distribution -- www.enthought.com Version: 7.1-1 (32-bit)

Python 2.7.2 |EPD 7.1-1 (32-bit)| (default, Jul 3 2011, 15:13:59) [MSC v.1500 32 bit (Intel)] on win32 Type "packages", "demo" or "enthought" for more information.

import scipy scipy.version '0.9.0'

Here are what I think are relevant part of output from 'nosetests -v statsmodels' :

---

... test_StepFunctionValueSideRight (statsmodels.distributions.tests.test_ecdf.TestDistributions) ... ok test_monotone_fn_inverter (statsmodels.distributions.tests.test_ecdf.TestDistributions) ... FAIL statsmodels.genmod.tests.test_glm.TestGaussianInverse.test_aic_R ... ok ... Test Baxter-King band-pass filter with 2d input ... ok Test Hodrick-Prescott Filter. Results taken from Stata. ... FAIL Test Christiano-Fitzgerald Filter. Results taken from R. ... ok ... statsmodels.tsa.vector_ar.tests.test_var.TestVARResultsLutkepohl.test_approx_mse ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResultsLutkepohl.test_cum_irf_stderr ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResultsLutkepohl.test_irf_stderr ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResultsLutkepohl.test_lr_effect_stderr ... ok statsmodels.tsa.vector_ar.tests.test_var.test_lutkepohl_parse ... ok statsmodels.tsa.vector_ar.tests.test_var.test_get_trendorder ... ok

---

Then the tests hang.

[josef-pkt]

When you do control+C when the tests hangs, do you see from the traceback (towards the end) what it was working on.

The only other way to find the hanging test, that I know is to run individual tests.

the next tests after this on my computer is test_ar, for example running a specific test file or specific test (uses file path)

nosetests -v path\to\statsmodels\tsa\tests\test_ar.py

nosetests -v path\to\statsmodels\tsa\tests\test_ar.py:TestARMLEConstant

Thanks for testing, we are then also interested in how or why the Test Hodrick-Prescott Filter fails

[breisfeld]

When I do control+C at the hang, I get the following:

forrtl: error (200): program aborting due to control-C event Image PC Routine Line Source mk2ifcoremd.dll 04654B8A Unknown Unknown Unknown mk2ifcoremd.dll 045CBE34 Unknown Unknown Unknown mk2ifcoremd.dll 045D9C88 Unknown Unknown Unknown kernel32.dll 751D6E51 Unknown Unknown Unknown kernel32.dll 7513339A Unknown Unknown Unknown ntdll.dll 77019EF2 Unknown Unknown Unknown ntdll.dll 77019EC5 Unknown Unknown Unknown

Both of the following hang and give the above 'traceback' when I hit control+C:

[C:..\tsa\tests]: nosetests -v test_ar.py [C:..\tsa\tests]: nosetests -v test_ar.py:TestARMLEConstant

The Hodrick-Prescott Filter test is more informative:

---

[C:..\filters\tests]:nosetests -v .\test_filters.py Test Baxter King band-pass filter. Results are taken from Stata ... ok Test Baxter-King band-pass filter with 2d input ... ok Test Hodrick-Prescott Filter. Results taken from Stata. ... FAIL Test Christiano-Fitzgerald Filter. Results taken from R. ... ok

FAIL: Test Hodrick-Prescott Filter. Results taken from Stata.

Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 187, in runTest self.test(*self.arg) File "C:..\statsmodels\tsa\filters\tests\test_filters.py", line 328, in test_hpfilter assert_almost_equal(res,hpfilt_res,6) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 452, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 800, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 6 decimals

(mismatch 100.0%) x: array([[ 0. , 2710.349], [ 0. , 2778.801], [ 0. , 2775.488],... y: array([[ 39.51191484, 2670.83708516], [ 80.08853246, 2698.71246754], [ 48.87545512, 2726.61254488],...

---

Ran 4 tests in 0.349s

FAILED (failures=1)

---

[josef-pkt]

My guess would be an endless loop in fmin_l_bfgs_b, but I don't think I have seen one there yet. fmin_l_bfgs_b uses fortran, and as far as I can see there is no extension used in AR.

We need a standalone example to debug this and try with different solvers.

To see if there are other problems in the tests, you could prevent this test from running by temporarily renaming the classname in the tests ( I usually just drop the T from Test)

class TestARMLEConstant(object): -> class estARMLEConstant(object):

or better try a different solver

class TestARMLEConstant(object):
    @classmethod
    def setupClass(cls):
        data = sm.datasets.sunspots.load()
-        cls.res1 = AR(data.endog).fit(maxlag=9,method="mle", disp=-1)
+        cls.res1 = AR(data.endog).fit(maxlag=9,method="mle", disp=-1, solver='nm')
        cls.res2 = results_ar.ARResultsMLE(constant=True)

When I do this I get a test failure because the numbers differ too much, but optimize.fmin shouldn't hang

[josef-pkt]

sorry if you got the last message many times, firefox hung and then posted it several times

[breisfeld]

I made the change to include the keyword "solver='nm'" in AR().fit() and there is no longer a hang, and as expected, I get the following:

[C:..\tsa\tests]:nosetests -v test_ar.py:TestARMLEConstant statsmodels.tsa.tests.test_ar.TestARMLEConstant.test_predict ... FAIL

FAIL: statsmodels.tsa.tests.test_ar.TestARMLEConstant.test_predict

Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 187, in runTest self.test(*self.arg) File "C:...\statsmodels\tsa\tests\test_ar.py", line 127, in test_predict DECIMAL_4) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 452, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 800, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 4 decimals

(mismatch 100.0%) x: array([ 48.39404075, 12.85630642, 26.70124621, 31.58246094, 36.64284541, 49.57934413, 68.19538626, 16.76383517, 13.73570649, 11.58576889, 9.5677924 , 9.45541006,... y: array([ 48.3243, 12.6633, 26.1958, 30.743 , 35.6854, 48.6206, 67.6159, 16.0018, 13.376 , 11.2305, 9.4877, 9.0588, 8.7644, 11.2268, 20.9667, 23.5576, 36.9585, 50.0107,...

---

Ran 1 test in 0.487s

FAILED (failures=1)

[josef-pkt]

Thanks, that's exactly the same as I have. If I use solver='bfgs', then I don't get a test failure. However, bfgs is also risking an endless loop if you don't have the fixed version of scipy (I don't remember right now when it was changed in scipy).

Did you manage to run the rest of the test suite?

[josef-pkt]

AR with default optimize.fmin_l_bfgs_b doesn't use maxiter, the optimizer doesn't have the argument. fmin_l_bfgs_b has maxfun=15000, but I don't know whether it would help to limit long/endless loops. (In scipy bfgs it was the line search that went into an endless loop, and not the iteration.)

[breisfeld]

If I run the full test suite I get another hang:

Here is the end of the verbose output:

... statsmodels.tsa.vector_ar.tests.test_var.TestVARResults.test_stderr ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResults.test_summary ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResults.test_tstat ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResultsLutkepohl.test_approx_mse ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResultsLutkepohl.test_cum_irf_stderr ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResultsLutkepohl.test_irf_stderr ... ok statsmodels.tsa.vector_ar.tests.test_var.TestVARResultsLutkepohl.test_lr_effect_stderr ... ok statsmodels.tsa.vector_ar.tests.test_var.test_lutkepohl_parse ... ok statsmodels.tsa.vector_ar.tests.test_var.test_get_trendorder ... ok

[josef-pkt]

That's the same spot as before. Did you change the solver in the installed directory?
If you want to change the tests in a different directory, then you could just run the tests with the directory path, path/to/statsmodels

Does nosetests -v path\to\statsmodels\tsa\tests\test_ar.py still hang with a second test?

[breisfeld]

Sorry... path issues. I still get a hang, but in a different spot:

... statsmodels.tsa.tests.test_ar.TestAutolagAR.test_ic ... ok statsmodels.tsa.tests.test_arima_process.test_fi ... ok statsmodels.tsa.tests.test_arima_process.test_arma_impulse_response ... ok statsmodels.tsa.tests.test_arima_process.test_spectrum ... ok statsmodels.tsa.tests.test_arima_process.test_armafft ... ok

[josef-pkt]

(I got distracted with other things)

The next test is statsmodels.tsa.tests.test_arma.Test_Y_ARMA02_Const which also uses fmin_l_bfgs_b. So do most other tests in test_arma.py

It looks to me that your fmin_l_bfgs_b is not working properly.

What you could try is to switch to solver='bfgs' as default in ARMA.fit, that is change the default value for the solver keyword in arima_model.py

    def fit(self, order, start_params=None, trend='c', method = "css-mle",
            transparams=True, solver=None, maxiter=35, full_output=1,
            disp=5, callback=None, **kwargs):

The problem is that in scipy 0.9 fmin_bfgs can also go into an endless loop if there are nans somewhere. fmin (Nelder-Mead) is safe but sometimes not precise enough. Upgrading scipy to 0.10 should solve the fmin_bfgs problems.

trying to limit the number of function evaluations, maxfun, which is 15000 in l_bfgs_b, might be worth a try, but wouldn't solve that it doesn't converge.

[jseabold]

Since this is a scipy bug can we close it?

[josef-pkt]

We can close it. We haven't heard or seen anything like this since then, nor have we been able to replicate it. (might have been an EPD bug)