search

GeoscienceAustralia / PyRate

A Python tool for estimating velocity and time-series from Interferometric Synthetic Aperture Radar (InSAR) data.
https://geoscienceaustralia.github.io/PyRate/
Apache License 2.0
199 stars 69 forks source link

Avoid costly file read/write step during orbital error correction #66

Closed basaks closed 8 years ago

basaks commented 8 years ago

instead just use in memory data, possibly by a different list of Ifg instances

basaks commented 8 years ago

This has been accomplished by # c848f78de21d50fb3b3cf360640e6b906e8c362c.

mcgarth commented 8 years ago

I think matlab_mst_kruskal_from_ifgs is expecting the multi-look objects to be an item on disk still:

(pyrate)[u49942@rhe-compute1 RABAUL]: python /nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_pyrate.py pyrate.conf Calculating minimum spanning tree matrix using NetworkX method Finding reference pixel Reference pixel coordinate: (574, 133) Calculating orbital error correction Traceback (most recent call last): File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_pyrate.py", line 437, in <module> main() File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_pyrate.py", line 392, in main process_ifgs(dest_paths, pars) File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_pyrate.py", line 53, in process_ifgs remove_orbital_error(ifgs, params) File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/scripts/run_pyrate.py", line 217, in remove_orbital_error mlooked=mlooked) File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/orbital.py", line 79, in orbital_correction _network_correction(ifgs, degree, offset, mlooked) File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/orbital.py", line 164, in _network_correction src_ifgs = matlab_mst.matlab_mst_kruskal_from_ifgs(src_ifgs) File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/matlab_mst_kruskal.py", line 296, in matlab_mst_kruskal_from_ifgs dest_paths = [i.data_path for i in ifgs] AttributeError: 'Ifg' object has no attribute 'data_path'

basaks commented 8 years ago

I have fixed this. There is a test case now for functionality check in pyrate/tests/test_orbital.py in test_orbital_error_method2_dummy.

mcgarth commented 8 years ago

I think you're right! 4b74f4b is giving me 1 error 3 failures as below on rhe-compute. Close if you like @basaks 

ERROR: test suite for <class 'pyrate.tests.test_prepifg_pypar.MyTestCase'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 209, in run
    self.setUp()
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/tests/test_prepifg_pypar.py", line 33, in setUpClass
    subprocess.check_call(cmd, shell=True)
  File "/nas/users/u49942/unix/anaconda2/envs/pyrate/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'mpirun -np 4 python pyrate/nci/run_prepifg_pypar.py /tmp/tmpppwLtp/tmpP0P2hA.conf' returned non-zero exit status 1

======================================================================
FAIL: test_basic_outputs (pyrate.tests.test_pyrate.PyRateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/tests/test_pyrate.py", line 172, in test_basic_outputs
    log_path = self.get_logfile_path()
  File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/tests/test_pyrate.py", line 156, in get_logfile_path
    self.fail(msg)
AssertionError: Log not generated. Use --nologcapture if running nosetests

======================================================================
FAIL: test_refpixel_found (pyrate.tests.test_pyrate.PyRateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/tests/test_pyrate.py", line 192, in test_refpixel_found
    log_path = self.get_logfile_path()
  File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/tests/test_pyrate.py", line 156, in get_logfile_path
    self.fail(msg)
AssertionError: Log not generated. Use --nologcapture if running nosetests

======================================================================
FAIL: test_readonly_permission_failure (pyrate.tests.test_shared.IfgIOTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nas/gemd/insar/Users/Garthwaite/repo/PyRate/pyrate/tests/test_shared.py", line 182, in test_readonly_permission_failure
    self.assertRaises(IOError, self.ifg.open, False)
AssertionError: IOError not raised

----------------------------------------------------------------------
Ran 251 tests in 126.368s

FAILED (errors=1, failures=3)
basaks commented 8 years ago

That FAIL: test_readonly_permission_failure (pyrate.tests.test_shared.IfgIOTests) is not a problem, just the file permission test. It goes away most times if you run it again.

Similarly ERROR: test suite for <class 'pyrate.tests.test_prepifg_pypar.MyTestCase'> this is ok too for reasons we have discussed on another issue.