search

GeoscienceAustralia / PyRate

A Python tool for estimating velocity and time-series from Interferometric Synthetic Aperture Radar (InSAR) data.
https://geoscienceaustralia.github.io/PyRate/
Apache License 2.0
203 stars 71 forks source link

Prepifg with PARALLEL=False and PROCESSOR=GAMMA returns roipac tifs after implementing restructure changes #214

Closed brenmous closed 5 years ago

brenmous commented 5 years ago

As part of implementing the prepifg restructure, I'm running and debugging tests, refactoring where needed. The test_gamma.TestGammaParallelVsSerial test fails on comparing gamma ifgs processed serially and in parallel.

The cause is strange. The output of the serial prepifg process is processed ROIPAC ifgs sitting in a temporary tif directory in the OBS_DIR, and the parallel ifgs are GAMMA and sit in the tmp output directory specified by the test as intended.

I've made some changes to the test to accommodate that prepifg no longer does geotiff conversion, and that the test ignores the comparison of DEM and incidence files. I'm fairly confident it's not the test causing the issues, as the exact same process is run with the only change being the setting of the PARALLEL argument. I've narrowed down to something in the prepifg module, as the paths are correct as the data flows through convertogtif and run_prepifg. Working on it now, will update.

https://github.com/GeoscienceAustralia/PyRate/compare/master...bren-restructure

SERIAL IFGS
[Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_060619-061002_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_060828-061211_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_061002-070219_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_061002-070430_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_061106-061211_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_061106-070115_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_061106-070326_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_061211-070709_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_061211-070813_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_070115-070326_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_070115-070917_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_070219-070430_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_070219-070604_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_070326-070917_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_070430-070604_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_070604-070709_unw.tif'),
 Ifg('/home/ubuntu/PyRate/tests/test_data/small_test/tif/geo_070709-070813_unw.tif')]

PARA IFGS
[Ifg('/tmp/tmp51m8rlu3/20060619-20061002_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20060828-20061211_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20061002-20070219_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20061002-20070430_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20061106-20061211_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20061106-20070115_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20061106-20070326_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20061211-20070709_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20061211-20070813_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20070115-20070326_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20070115-20070917_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20070219-20070430_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20070219-20070604_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20070326-20070917_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20070430-20070604_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20070604-20070709_utm_unw_1rlks_1cr.tif'),
 Ifg('/tmp/tmp51m8rlu3/20070709-20070813_utm_unw_1rlks_1cr.tif')]
brenmous commented 5 years ago

The ROIPAC tifs is because the test_common.small_data_setup method returns those tifs by default if no datafiles are provided (otherwise it opens them as Ifgs and sets the nodata value to 0). It gets called on the prepared IFGs in the test before the comparison is performed. So the problem isn't that serial gamma prepifg isn't preparing ROIPAC ifgs, but not preparing ifgs at all.

brenmous commented 5 years ago

Aaaaaaand it was a typo. I changed the glob for finding the prepared IFGs to "*utm_unw_1rlks_1cr.tif" to exclude the DEM and Incidence files, and misspelt this suffix when globbing the serial files. Nothing to see here, move along.