Tests not passing - environment differences?

[mcgarth]

Sudipta reports a full suite of passing tests, though when I run the full test suite on rhe-compute1 I get failures in test_timeseries (6 failures) related to nan location mismatches in matrices and 2 errors attributable to a linalg error in linrate tests. It also runs very slowly - maybe 2-3 minutes to run the full test suite. Code is the same so more likely a difference in anaconda environment?

Attached is my anaconda environment list.

conda_list_matt.txt

[basaks]

New anaconda environment_simple.yml added. Please pull and create new environment on rhe-compute.

I ran nosetests with the following results:

Ran 230 tests in 41.891s
FAILED (failures=2)

The two fails are the nologcaputure ones.

[mcgarth]

I have created the new conda environment and run the tests on rhe-compute - MUCH BETTER!

Ran 228 tests in 99.542s FAILED (errors=1, failures=2)

The 2 fails are the nologcapture ones also. The error is:

ERROR: test suite for <class 'pyrate.tests.test_prepifg_pypar.MyTestCase'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 209, in run
    self.setUp()
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/tests/test_prepifg_pypar.py", line 33, in setUpClass
    subprocess.check_call(cmd, shell=True)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'mpirun -np 4 python pyrate/nci/run_prepifg_pypar.py /tmp/tmpolPptu/tmpc_ADxZ.conf' returned non-zero exit status 1

[basaks]

This is really good. This means all the existing tests pass for you. The 3rd one that is failing is my latest addition and is failing due to some mpi config issues on rhe-compute. It works on my pc.

On the other hand, why does rhe-compute nosetests find 2 fewer tests than on my pc (230 compared to 228)? Will have to check this too, but possibly much lower priority. Edit: This is because the mpirun setup needs to be checked on rhe-compute. The pyrate.tests.test_prepifg_pypar has 2 tests which are not being detected because test setUp is failing.

Close this if you are happy :)

[mcgarth]

update 27 April: still only picking up 228 tests!

[basaks]

@mcgarth Have you export PYTHONPATH=$PYRATEPATH:$PYTHONPATH? Normally this is not necessary for nosetests, but this is necessary due to the subprocess call in the mpirun test.

The pyrate.tests.test_prepifg_pypar is likely failing for you as you have not exported this and modules are not found by the suprocess call. This is causing test setUp to fail on that test file, and the two tests in that file are not being found by nosetests.

These tests are even working on windows using anaconda.

I don't have access to rhe-compute at the moment to test.

[mcgarth]

Thanks @basaks - if using the anaconda environment, what should PYTHONPATH be before this export command?

[basaks]

When yo activate your anaconda environment, the python path is added to your env variables. You just need to do this:

activate your anaconda environment cd PyRate ( pyrate clone dir) export PYRATEPATH=$(pwd) export PYTHONPATH=$PYRATEPATH:$PYTHONPATH

[mcgarth]

(pyrate)[u49942@rhe-compute1 PyRate]: echo $PYTHONPATH /nas/users/u49942/unix/anaconda2/envs/pyrate/bin/:/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate (pyrate)[u49942@rhe-compute1 PyRate]: echo $PYRATEPATH /nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate

Not sure this is the issue. Still get the test_prepifg_pypar error:

ERROR: test suite for <class 'pyrate.tests.test_prepifg_pypar.MyTestCase'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 209, in run
    self.setUp()
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/tests/test_prepifg_pypar.py", line 33, in setUpClass
    subprocess.check_call(cmd, shell=True)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'mpirun -np 4 python pyrate/nci/run_prepifg_pypar.py /tmp/tmpgqmLIh/tmpJedxbZ.conf' returned non-zero exit status 1```

[basaks]

This command works for you?

mpirun -np 4 python pyrate/nci/run_prepifg_pypar.py pyrate_gamma.conf

If it does not, can you give me the traceback?

[mcgarth]

Nope :-( It appears mpi is not properly compiled

ERROR: C extension mpiext could not be imported.
Please compile mpiext.c e.g. by running
  python compile_pypar_locally.py
in the pypar directory, or by using
  python setup.py install

WARNING: MPI library could not be initialised. Running sequentially
Master process found 1 worker processors
ERROR: C extension mpiext could not be imported.
Please compile mpiext.c e.g. by running
  python compile_pypar_locally.py
in the pypar directory, or by using
  python setup.py install

WARNING: MPI library could not be initialised. Running sequentially
Master process found 1 worker processors
ERROR: C extension mpiext could not be imported.
Please compile mpiext.c e.g. by running
  python compile_pypar_locally.py
in the pypar directory, or by using
  python setup.py install

WARNING: MPI library could not be initialised. Running sequentially
Master process found 1 worker processors
ERROR: C extension mpiext could not be imported.
Please compile mpiext.c e.g. by running
  python compile_pypar_locally.py
in the pypar directory, or by using
  python setup.py install

WARNING: MPI library could not be initialised. Running sequentially
Master process found 1 worker processors
Processor 0 has 17 sites out of 17
running gamma prepifg
Processor 0 has 17 sites out of 17
running gamma prepifg
Processor 0 has 17 sites out of 17
running gamma prepifg
Processor 0 has 17 sites out of 17
running gamma prepifg
Traceback (most recent call last):
  File "pyrate/nci/run_prepifg_pypar.py", line 108, in <module>
    main()
  File "pyrate/nci/run_prepifg_pypar.py", line 90, in main
Traceback (most recent call last):
  File "pyrate/nci/run_prepifg_pypar.py", line 108, in <module>
    [run_prepifg.gamma_multiprocessing(b, params) for b in process_base_paths]
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_prepifg.py", line 123, in gamma_multiprocessing
    main()
  File "pyrate/nci/run_prepifg_pypar.py", line 90, in main
    DEM_HDR = gamma.parse_dem_header(dem_hdr_path)
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/gamma.py", line 85, in parse_dem_header
    [run_prepifg.gamma_multiprocessing(b, params) for b in process_base_paths]
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_prepifg.py", line 136, in gamma_multiprocessing
    lookup = parse_header(path)
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/gamma.py", line 50, in parse_header
    write_geotiff(COMBINED, b, d, nodata=params[cf.NO_DATA_VALUE])
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/shared.py", line 563, in write_geotiff
    raw_segs = [line.split() for line in text if ':' in line]
RuntimeError: TIFFResetField:out/20060619-20061002_utm.tif: Could not find tag 273.
Traceback (most recent call last):
  File "pyrate/nci/run_prepifg_pypar.py", line 108, in <module>
    ds = driver.Create(dest, ncols, nrows, 1, dtype)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/osgeo/gdal.py", line 394, in Create
    main()
  File "pyrate/nci/run_prepifg_pypar.py", line 90, in main
    return _gdal.Driver_Create(self, _args, *_kwargs)
RuntimeError: Unable to open out/20060619-20061002_utm.tif to obtain file list.
    [run_prepifg.gamma_multiprocessing(b, params) for b in process_base_paths]
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_prepifg.py", line 136, in gamma_multiprocessing
    write_geotiff(COMBINED, b, d, nodata=params[cf.NO_DATA_VALUE])
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/shared.py", line 563, in write_geotiff
    ds = driver.Create(dest, ncols, nrows, 1, dtype)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/osgeo/gdal.py", line 394, in Create
    return _gdal.Driver_Create(self, _args, *_kwargs)
RuntimeError: TIFFReadDirectory:Failed to read directory at offset 14246
Warning 1: TIFFFetchNormalTag:IO error during reading of "GeoPixelScale"; tag ignored
Warning 1: TIFFFetchNormalTag:IO error during reading of "GeoTiePoints"; tag ignored
Warning 1: TIFFFetchNormalTag:IO error during reading of "GeoKeyDirectory"; tag ignored
Warning 1: TIFFFetchNormalTag:IO error during reading of "GeoDoubleParams"; tag ignored
Warning 1: TIFFFetchNormalTag:IO error during reading of "GeoASCIIParams"; tag ignored
Warning 1: TIFFFetchNormalTag:IO error during reading of "GDALMetadata"; tag ignored
Traceback (most recent call last):
  File "pyrate/nci/run_prepifg_pypar.py", line 108, in <module>
    main()
  File "pyrate/nci/run_prepifg_pypar.py", line 90, in main
    [run_prepifg.gamma_multiprocessing(b, params) for b in process_base_paths]
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_prepifg.py", line 136, in gamma_multiprocessing
    write_geotiff(COMBINED, b, d, nodata=params[cf.NO_DATA_VALUE])
  File "/nas/users/u49942/unix/insar/Users/Garthwaite/repo/PyRate/pyrate/shared.py", line 563, in write_geotiff
    ds = driver.Create(dest, ncols, nrows, 1, dtype)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/osgeo/gdal.py", line 394, in Create
    return _gdal.Driver_Create(self, _args, *_kwargs)

## RuntimeError: Line 0: Didn't find expected '=' for value of attribute 'ßk'.

mpirun noticed that the job aborted, but has no info as to the process

## that caused that situation.

(pyrate)[u49942@rhe-compute1 PyRate]:```

[basaks]

Well your mpi has got issues, but that is not why the test is failing. The package recovers from the mpi failure, and tries to run in sequential mode. The problem seems to be with anaconda gdal installation:

File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/osgeo/gdal.py", line 394, in Create
return _gdal.Driver_Create(self, args, *kwargs)

[mcgarth]

I am not sure why but this seems to have been resolved by the changes up to commit 93b0529. I get 235 tests running with only two "log not generated" failures.

I'll close now before it breaks again ;-)