The error below interrupted the workflow but running again they worked fine. If these are freak errors caused by the system, can we catch them and run the job again if they have appeared?
Error: check_job_outputs.py /scratch/05861/tg851601/KashgarSenDT34/run_files/run_06_overlap_resample_0.job
For known issues see https://github.com/geodesymiami/rsmas_insar/tree/master/docs/known_issues.md
Traceback (most recent call last):
File "/scratch/05861/tg851601/code1/rsmas_insar/minsar/check_job_outputs.py", line 70, in <module>
main()
File "/scratch/05861/tg851601/code1/rsmas_insar/minsar/check_job_outputs.py", line 54, in main
raise RuntimeError('Error: \"' + error_string + '\" found in ' + file)
RuntimeError: Error: "Error" found in /scratch/05861/tg851601/KashgarSenDT34/run_files/run_06_overlap_resample_0_20151202_14.e
check_job_outputs.py /scratch/05861/tg851601/KashgarSenDT34/run_files/run_06_overlap_resample_0.job exited with a non-zero exit code (1). Exiting.
cat run_06_overlap_resample_0_20151202_14.e
--- Logging error ---
Traceback (most recent call last):
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/logging/handlers.py", line 70, in emit
self.doRollover()
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/logging/handlers.py", line 167, in doRollover
os.rename(sfn, dfn)
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/05861/tg851601/KashgarSenDT34/isce.log.3' -> '/scratch/05861/tg851601/KashgarSenDT34/isce.log.4'
Call stack:
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 251, in <module>
main(args.start,args.end)
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 242, in main
cfgParser.runCmd()
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 54, in runCmd
func_modules.main(self.funcParams[section])
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/resamp_withCarrier.py", line 236, in main
outimg = resampSecondary(botBurst, slvBurst, rdict, outname, (not inps.noflat))
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/resamp_withCarrier.py", line 95, in resampSecondary
rObj.resamp_slc(imageOut=imgOut)
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/stdproc/stdproc/resamp_slc/Resamp_slc.py", line 75, in resamp_slc
self.setDefaults()
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/stdproc/stdproc/resamp_slc/Resamp_slc.py", line 182, in setDefaults
self.logger.warning('The variable INPUT_LINES has been set to the default value %d which is the number of lines in the slc image.' % (self.inputLines))
Message: 'The variable INPUT_LINES has been set to the default value 1499 which is the number of lines in the slc image.'
Arguments: ()
Error: check_job_outputs.py /scratch/05861/tg851601/KashgarSenAT129/run_files/run_06_overlap_resample_2.job
For known issues see https://github.com/geodesymiami/rsmas_insar/tree/master/docs/known_issues.md
Traceback (most recent call last):
File "/scratch/05861/tg851601/code/rsmas_insar/minsar/check_job_outputs.py", line 70, in <module>
main()
File "/scratch/05861/tg851601/code/rsmas_insar/minsar/check_job_outputs.py", line 54, in main
raise RuntimeError('Error: \"' + error_string + '\" found in ' + file)
RuntimeError: Error: "Error" found in /scratch/05861/tg851601/KashgarSenAT129/run_files/run_06_overlap_resample_2_20190521_5.e
check_job_outputs.py /scratch/05861/tg851601/KashgarSenAT129/run_files/run_06_overlap_resample_2.job exited with a non-zero exit code (1). Exiting.
//login3/scratch/05861/tg851601/KashgarSenAT129[1007] cat /scratch/05861/tg851601/KashgarSenAT129/run_files/run_06_overlap_resample_2_20190521_5.e
--- Logging error ---
Traceback (most recent call last):
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/logging/handlers.py", line 70, in emit
self.doRollover()
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/logging/handlers.py", line 167, in doRollover
os.rename(sfn, dfn)
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/05861/tg851601/KashgarSenAT129/isce.log.4' -> '/scratch/05861/tg851601/KashgarSenAT129/isce.log.5'
Call stack:
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 251, in <module>
main(args.start,args.end)
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 242, in main
cfgParser.runCmd()
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 54, in runCmd
func_modules.main(self.funcParams[section])
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/resamp_withCarrier.py", line 210, in main
outimg = resampSecondary(topBurst, slvBurst, rdict, outname, (not inps.noflat))
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/resamp_withCarrier.py", line 95, in resampSecondary
rObj.resamp_slc(imageOut=imgOut)
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/stdproc/stdproc/resamp_slc/Resamp_slc.py", line 75, in resamp_slc
self.setDefaults()
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/stdproc/stdproc/resamp_slc/Resamp_slc.py", line 182, in setDefaults
self.logger.warning('The variable INPUT_LINES has been set to the default value %d which is the number of lines in the slc image.' % (self.inputLines))
Message: 'The variable INPUT_LINES has been set to the default value 1495 which is the number of lines in the slc image.'
Arguments: ()
Error: check_job_outputs.py /scratch/05861/tg851601/KashgarSenDT34/run_files/run_06_overlap_resample_1.job [186/291]
For known issues see https://github.com/geodesymiami/rsmas_insar/tree/master/docs/known_issues.md
Traceback (most recent call last):
File "/scratch/05861/tg851601/code/rsmas_insar/minsar/check_job_outputs.py", line 70, in <module>
main()
File "/scratch/05861/tg851601/code/rsmas_insar/minsar/check_job_outputs.py", line 54, in main
raise RuntimeError('Error: \"' + error_string + '\" found in ' + file)
RuntimeError: Error: "Error" found in /scratch/05861/tg851601/KashgarSenDT34/run_files/run_06_overlap_resample_1_20170712_8.e
check_job_outputs.py /scratch/05861/tg851601/KashgarSenDT34/run_files/run_06_overlap_resample_1.job exited with a non-zero exit code (1). Exiting.
//login3/scratch/05861/tg851601/KashgarSenDT34[1003]
//login3/scratch/05861/tg851601/KashgarSenDT34[1003]
//login3/scratch/05861/tg851601/KashgarSenDT34[1003]
//login3/scratch/05861/tg851601/KashgarSenDT34[1003] cat run_files/run_06_overlap_resample_1_20170712_8.e
--- Logging error ---
Traceback (most recent call last):
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/logging/handlers.py", line 70, in emit
self.doRollover()
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/logging/handlers.py", line 167, in doRollover
os.rename(sfn, dfn)
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/05861/tg851601/KashgarSenDT34/isce.log.4' -> '/scratch/05861/tg851601/KashgarSenDT34/isce.log.5'
Call stack:
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 251, in <module>
main(args.start,args.end)
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 242, in main
cfgParser.runCmd()
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 54, in runCmd
func_modules.main(self.funcParams[section])
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/resamp_withCarrier.py", line 236, in main
outimg = resampSecondary(botBurst, slvBurst, rdict, outname, (not inps.noflat))
File "/scratch/05861/tg851601/code/rsmas_insar/sources/isce2/contrib/stack/topsStack/resamp_withCarrier.py", line 95, in resampSecondary
rObj.resamp_slc(imageOut=imgOut)
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/stdproc/stdproc/resamp_slc/Resamp_slc.py", line 75, in resamp_
slc
self.setDefaults()
File "/scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/stdproc/stdproc/resamp_slc/Resamp_slc.py", line 182, in setDef
aults
self.logger.warning('The variable INPUT_LINES has been set to the default value %d which is the number of lines in the slc image.' % (self.inputLines))
Message: 'The variable INPUT_LINES has been set to the default value 1499 which is the number of lines in the slc image.'
Arguments: ()
//login3/scratch/05861/tg851601/KashgarSenDT34[1004] ll isce*
-rw-rw---- 1 tg851601 G-820134 28308 Sep 13 22:57 isce.log
-rw-rw---- 1 tg851601 G-820134 27971 Sep 13 22:56 isce.log.1
-rw-rw---- 1 tg851601 G-820134 25949 Sep 13 22:55 isce.log.2
-rw-rw---- 1 tg851601 G-820134 29319 Sep 13 22:58 isce.log.3
-rw-rw---- 1 tg851601 G-820134 29319 Sep 13 22:58 isce.log.4
-rw-rw---- 1 tg851601 G-820134 28308 Sep 13 22:56 isce.log.5
//login3/scratch/05861/tg851601/KashgarSenDT34[1005] vi /scratch/05861/tg851601/code/rsmas_insar/3rdparty/miniconda3/lib/python3.7/logging/handlers.py
//login3/scratch/05861/tg851601/KashgarSenDT34[1006] cat isce.log.3
2020-09-13 22:26:46,924 - isce.stdproc.resamp_slc - WARNING - The variable INPUT_LINES has been set to the default value 1499 which is the number of lines in the slc image.
2020-09-13 22:26:46,932 - isce.stdproc.resamp_slc - WARNING - The variable INPUT_WIDTH has been set to the default value 21865 which is the width of the slc image.
falkamelung
commented
3 years ago
The error below interrupted the workflow but running again they worked fine. If these are freak errors caused by the system, can we catch them and run the job again if they have appeared?