search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
59 stars 23 forks source link

run_10_filter_coherence missing fine.int.xml errors #479

Open falkamelung opened 3 years ago

falkamelung commented 3 years ago

KokoxiliBigChunk34SenAT143 grep FileNotFoundError out_run_10_filter_coherence.e 

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20161121_20170225/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20161215_20170108/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20161215_20170213/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20190919_20191106/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20190919_20191118/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20191001_20191013/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20191001_20191025/fine.int.xml'
Error: "FileNotFoundError" found in /scratch/05861/tg851601/KokoxiliBigChunk34SenAT143/run_files/run_10_filter_coherence_29_20190919_20191106_8.e
Error: "FileNotFoundError" found in /scratch/05861/tg851601/KokoxiliBigChunk34SenAT143/run_files/run_10_filter_coherence_29_20190919_20191118_9.e
Error: "FileNotFoundError" found in /scratch/05861/tg851601/KokoxiliBigChunk34SenAT143/run_files/run_10_filter_coherence_29_20191001_20191013_10.e
Error: "FileNotFoundError" found in /scratch/05861/tg851601/KokoxiliBigChunk34SenAT143/run_files/run_10_filter_coherence_29_20191001_20191025_11.e
Error: "FileNotFoundError" found in /scratch/05861/tg851601/KokoxiliBigChunk34SenAT143/run_files/run_10_filter_coherence_29_20191001_20191106_12.e
Error: "FileNotFoundError" found in /scratch/05861/tg851601/KokoxiliBigChunk34SenAT143/run_files/run_10_filter_coherence_29_20191013_20191118_17.e
Error: "FileNotFoundError" found in /scratch/05861/tg851601/KokoxiliBigChunk34SenAT143/run_files/run_10_filter_coherence_36_20200715_20200913_1.e
Error: "FileNotFoundError" found in /scratch/05861/tg851601/KokoxiliBigChunk34SenAT143/run_files/run_10_filter_coherence_36_20200727_20200808_2.e

There are files missing that should have been produced by the previous step. 

ll merged/interferograms/20161121_20170225/
total 6280
-rw-rw---- 1 tg851601 G-820134 6398928 May  7 16:35 fine.int
-rw-rw---- 1 tg851601 G-820134   12683 May  7 16:33 fine.int.full.vrt
-rw-rw---- 1 tg851601 G-820134    4024 May  7 16:33 fine.int.full.xml

It should have show:

ll merged/interferograms/*
merged/interferograms/20160605_20160629:
total 6048
-rw-rw---- 1 tg851601 G-820134 6165280 May  9 04:56 fine.int
-rw-rw---- 1 tg851601 G-820134    1088 May  9 04:56 fine.int.full.vrt
-rw-rw---- 1 tg851601 G-820134    4021 May  9 04:56 fine.int.full.xml
-rw-rw---- 1 tg851601 G-820134     466 May  9 04:56 fine.int.vrt
-rw-rw---- 1 tg851601 G-820134    4246 May  9 04:56 fine.int.xml

Why did run_09_merge_burst_igram did not producefine.int.vrt and fine.int.xml, without producing errors? There were time outs. Does rerunning of run_09_merge_burst_igram not work after timeout?

I did check using just 1 job file and rerunning after timeout including producing the missing files worked fine.

I have lengthened the walltime. If this continues to happen we have to count files after computing completes and thrwo an error if files are missing.

falkamelung commented 3 years ago

MakranBigSenAT13 : another case with the same problem. The error shows up in a run_10 job.

grep FileNotFoundError out_run_10_filter_coherence.e 
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180705_20180810/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180705_20180822/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180717_20180729/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180717_20180810/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180717_20180822/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180717_20180903/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180729_20180810/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180729_20180822/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180729_20180903/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180729_20180915/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180810_20180822/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180810_20180903/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180810_20180915/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20180810_20180927/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20181114_20181126/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20181114_20181208/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20181114_20181220/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20181126_20181208/fine.int.xml'
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged/interferograms/20181126_20190113/fine.int.xml'