search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
58 stars 22 forks source link

Errors in PYSAR? #107

Closed Ovec8hkin closed 5 years ago

Ovec8hkin commented 5 years ago

The following error shows up in the log files produced by process_rsmas when called from run_operations. This does not occur for all datasets, but it does occur consistently for the datasets for which it occurs. I don't know if this is a PySAR issue, a process_rsmas issue, or a run_operations issue.

PySAR version 1.0.0-dev, release date 2019-03-16
--2019-03-27 11:23:23.954413--
Run routine processing with pysarApp.py on steps: ['load_data']
--------------------------------------------------
Project name: testLombokSenAT156
create directory: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR
Go to work directory: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR
copy default template file /nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/defaults/pysarApp_template.txt to work directory
create directory: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/INPUTS
copy testLombokSenAT156.template to INPUTS directory for backup.
read custom template file: /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template
update default template based on input custom template
    pysar.load.processor: auto --> isce
    pysar.reference.lalo: auto --> -8.6456,116.5234
    pysar.networkInversion.weightFunc: auto --> fim
    pysar.troposphericDelay.method: auto --> no
    pysar.save.hdfEos5: auto --> yes
    pysar.save.hdfEos5.update: auto --> yes
read default template file: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt

******************** step - load_data ********************
load_data.py --template /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template --project testLombokSenAT156
platform : Sen
processor: isce
check auto path setting for Univ of Miami users for processor: isce
--------------------------------------------------
prepare metadata files for isce products for PySAR
--------------------------------------------------
searching interferometric pairs info
input data files:
WARNING: No reqired unwrapPhase data files found!
--------------------------------------------------
searching geometry files info
input data files:
WARNING: No reqired height data files found!
--------------------------------------------------
updateMode : True
compression: None
Traceback (most recent call last):
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/pysarApp.py", line 1021, in <module>
    main()
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/pysarApp.py", line 1011, in main
    app.run(steps=inps.runSteps, plot=inps.plot)
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/pysarApp.py", line 940, in run
    self.run_load_data(sname)
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/pysarApp.py", line 337, in run_load_data
    load_complete, stack_file, geom_file = ut.check_loaded_dataset(self.workDir, print_msg=True)[0:3]
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/utils/utils.py", line 70, in check_loaded_dataset
    raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), './INPUTS/ifgramStack.h5')
FileNotFoundError: [Errno 2] No such file or directory: './INPUTS/ifgramStack.h5'
2019-03-27 11:23:24,243 - INFO - pysarApp.py /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template

_________________________________________________
       ____             __     __     ____  
       /    )         /    )   / |    /    )
------/____/----------\-------/__|---/___ /------
     /        /   /    \     /   |  /    |  
____/________(___/_(____/___/____|_/_____|_______
                /                           
            (_ /                            

 A Python package for InSAR time series analysis.
           PySAR v1.0.0-dev, 2019-03-16
_________________________________________________

--2019-03-27 11:23:25.642668--
Run routine processing with pysarApp.py on steps: ['load_data', 'reference_point', 'stack_interferograms', 'correct_unwrap_error', 'modify_network', 'invert_network', 'correct_LOD', 'correct_troposphere', 'deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
--------------------------------------------------
Project name: testLombokSenAT156
Go to work directory: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR
['/projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/INPUTS/testLombokSenAT156.template'] exists and is newer than ['/nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template'] --> skip.
read custom template file: /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template
update default template based on input custom template
No new option value found, skip updating /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt
read default template file: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt

******************** step - load_data ********************
load_data.py --template /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template --project testLombokSenAT156
platform : Sen
processor: isce
check auto path setting for Univ of Miami users for processor: isce
--------------------------------------------------
prepare metadata files for isce products for PySAR
--------------------------------------------------
searching interferometric pairs info
input data files:
WARNING: No reqired unwrapPhase data files found!
--------------------------------------------------
searching geometry files info
input data files:
WARNING: No reqired height data files found!
--------------------------------------------------
updateMode : True
compression: None
Traceback (most recent call last):
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/rsmas_isce/process_rsmas.py", line 143, in <module>
    process(inps)
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/rsmas_isce/process_rsmas.py", line 109, in process
    prs.run_pysar(inps, start_time)
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/rsmas_isce/_processSteps.py", line 407, in run_pysar
    putils.email_pysar_results(time_str, inps.custom_template)
  File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/rsmas_isce/_process_utilities.py", line 373, in email_pysar_results
    fList = glob.glob(prefix+'/'+fname)
UnboundLocalError: local variable 'prefix' referenced before assignment
yunjunz commented 5 years ago

I can not tell where or when it went wrong, could you share more information @Ovec8hkin ? PySAR stops when there is an error, but it could went wrong earlier already.

falkamelung commented 5 years ago

There are a few data sets which have the slcDir specified (slcDir=$SCRATCHDIR/AgungSenDT32/SLC). This may not be supported anymore. Just disable or delete those,

                                                    $SCRATCHDIR/AgungSenDT32/SLC

Falk Amelung Professor Department of Marine Geosciences Rosenstiel School of Marine and Atmospheric Sciences University of Miami 4600 Rickenbacker Causeway Miami, FL 33149 USA Tel: 305 421 4949 E-mail: famelung@rsmas.miami.edumailto:famelung@rsmas.miami.edu Web: http://insar.rsmas.miami.edu InSAR data: http://insarmaps.miami.edu

On Mar 27, 2019, at 2:14 PM, Joshua Zahner notifications@github.com<mailto:notifications@github.com> wrote:

The following error shows up in the log files produced by process_rsmas when called from run_operations. This does not occur for all datasets, but it does occur consistently for the datasets for which it occurs. I don't know if this is a PySAR issue, a process_rsmas issue, or a run_operations issue.

PySAR version 1.0.0-dev, release date 2019-03-16 --2019-03-27 11:23:23.954413-- Run routine processing with pysarApp.py on steps: ['load_data']

Project name: testLombokSenAT156 create directory: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR Go to work directory: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR copy default template file /nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/defaults/pysarApp_template.txt to work directory create directory: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/INPUTS copy testLombokSenAT156.template to INPUTS directory for backup. read custom template file: /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template update default template based on input custom template pysar.load.processor: auto --> isce pysar.reference.lalo: auto --> -8.6456,116.5234 pysar.networkInversion.weightFunc: auto --> fim pysar.troposphericDelay.method: auto --> no pysar.save.hdfEos5: auto --> yes pysar.save.hdfEos5.update: auto --> yes read default template file: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt

**** step - load_data **** load_data.py --template /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template --project testLombokSenAT156 platform : Sen processor: isce check auto path setting for Univ of Miami users for processor: isce

prepare metadata files for isce products for PySAR

searching interferometric pairs info input data files: WARNING: No reqired unwrapPhase data files found!

searching geometry files info input data files: WARNING: No reqired height data files found!

updateMode : True compression: None Traceback (most recent call last): File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/pysarApp.py", line 1021, in main() File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/pysarApp.py", line 1011, in main app.run(steps=inps.runSteps, plot=inps.plot) File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/pysarApp.py", line 940, in run self.run_load_data(sname) File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/pysarApp.py", line 337, in run_load_data load_complete, stack_file, geom_file = ut.check_loaded_dataset(self.workDir, print_msg=True)[0:3] File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/PySAR/pysar/utils/utils.py", line 70, in check_loaded_dataset raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), './INPUTS/ifgramStack.h5') FileNotFoundError: [Errno 2] No such file or directory: './INPUTS/ifgramStack.h5' 2019-03-27 11:23:24,243 - INFO - pysarApp.py /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template

   ____             __     __     ____
   /    )         /    )   / |    /    )

------/__/-----------------/|---/ /------ / / / \ / | / | /____(/_(//____|/____| / (_ /

A Python package for InSAR time series analysis. PySAR v1.0.0-dev, 2019-03-16

--2019-03-27 11:23:25.642668-- Run routine processing with pysarApp.py on steps: ['load_data', 'reference_point', 'stack_interferograms', 'correct_unwrap_error', 'modify_network', 'invert_network', 'correct_LOD', 'correct_troposphere', 'deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']

Project name: testLombokSenAT156 Go to work directory: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR ['/projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/INPUTS/testLombokSenAT156.template'] exists and is newer than ['/nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template'] --> skip. read custom template file: /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template update default template based on input custom template No new option value found, skip updating /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt read default template file: /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt

**** step - load_data **** load_data.py --template /projects/scratch/insarlab/jaz101/testLombokSenAT156/PYSAR/pysarApp_template.txt /nethome/jaz101/insarlab/OPERATIONS/TEMPLATES/testLombokSenAT156.template --project testLombokSenAT156 platform : Sen processor: isce check auto path setting for Univ of Miami users for processor: isce

prepare metadata files for isce products for PySAR

searching interferometric pairs info input data files: WARNING: No reqired unwrapPhase data files found!

searching geometry files info input data files: WARNING: No reqired height data files found!

updateMode : True compression: None Traceback (most recent call last): File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/rsmas_isce/process_rsmas.py", line 143, in process(inps) File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/rsmas_isce/process_rsmas.py", line 109, in process prs.run_pysar(inps, start_time) File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/rsmas_isce/_processSteps.py", line 407, in run_pysar putils.email_pysar_results(time_str, inps.custom_template) File "/nethome/jaz101/test/testfalk/rsmas_insar/sources/rsmas_isce/_process_utilities.py", line 373, in email_pysar_results fList = glob.glob(prefix+'/'+fname) UnboundLocalError: local variable 'prefix' referenced before assignment

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_isce%2Fissues%2F107&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C594305ba7401494a4b9a08d6b2f0da5d%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636893144868794305&sdata=LviLuq7iYO6LOGfzrxfZq5ycR5kf7gnro%2BUNlXs9BKs%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKpT3JXadszz9vMJNCIodGdYNXKdvWbqks5va9EzgaJpZM4cOsMw&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C594305ba7401494a4b9a08d6b2f0da5d%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636893144868804309&sdata=4TooLZWjfvXVKpnLrTWjSsCphS2Cv9iHFRXnF1ZwNrE%3D&reserved=0.

Ovec8hkin commented 5 years ago

@falkamelung that would be the case. Do we need to handle the case where the SLC directory is specified, or no? Should I just remove the specified SLC directory, or the whole dataset?

falkamelung commented 5 years ago

Just disable the dataset. Don’t we have flag for that? I will deal with this when I am back. It uses downloaded data from a different template file. This is of course a weird case which is difficult to support. The way to do it ti run the previous dataset with a different cleanopt (keeeing the SLCs) and remove the data when this dataset is processed, but I have to check whether cleanopt works well enough for this.

Falk Amelung Professor Department of Marine Geosciences Rosenstiel School of Marine and Atmospheric Sciences University of Miami 4600 Rickenbacker Causeway Miami, FL 33149 USA Tel: 305 421 4949 E-mail: famelung@rsmas.miami.edumailto:famelung@rsmas.miami.edu Web: http://insar.rsmas.miami.edu InSAR data: http://insarmaps.miami.edu

On Mar 27, 2019, at 3:35 PM, Joshua Zahner notifications@github.com<mailto:notifications@github.com> wrote:

@falkamelunghttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffalkamelung&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Ce1346385fa884c656a2008d6b2fc2d81%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636893193498865462&sdata=1s6TIHqcctPrPYvbCrnQ1PGL5RtQl2%2BQXEurt8btc0o%3D&reserved=0 that would be the case. Do we need to handle the case where the SLC directory is specified, or no? Should I just remove the specified SLC directory, or the whole dataset?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_isce%2Fissues%2F107%23issuecomment-477356857&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Ce1346385fa884c656a2008d6b2fc2d81%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636893193498875471&sdata=%2FWOXo1tPqAK041QgtAyfDc5OUtPwAKcp1OKot1RmhYI%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKpT3C61nqnnRAQkTHbf0YrkIpmi18_Rks5va-QzgaJpZM4cOsMw&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Ce1346385fa884c656a2008d6b2fc2d81%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636893193498875471&sdata=Bh6cRQKelT6eNvLCJFBsPHCEdz9oyBNa8tYVlAy7Nw4%3D&reserved=0.

Ovec8hkin commented 5 years ago

Alright. I disabled them. With that, I believe run_operations works as advertised! I will submit it for code review and go over it with @2gotgrossman later this week or this weekend. I will let you know when its merged and ready for your to test out.