missing pairs in `run_14_mergeBurstsIon` step

[yunjunz]

I am running the topsStack with ionospheric corrections, and had the following error in the "run_19_invertIon" step:

Traceback (most recent call last):
  File "/home/zyunjun/tools/isce2/src/isce2/contrib/stack/topsStack/invertIon.py", line 223, in <module>
    ionPairs[i, :, :] = (np.fromfile(ionfile, dtype=np.float32).reshape(length*2, width))[1:length*2:2, :]
FileNotFoundError: [Errno 2] No such file or directory: '/marmot-nobak/zyunjun/offset4motion/SaltonSeaSenDT173/ion/20150801_20150918/ion_cal/filt.ion'

The missing files originated from the earlier step run_14_mergeBurstsIon, where for some pairs, the lower/upper band merged interferogram files are not generated, check below for one pair with missing files and one pair with all the files. But when I check the log file / print out message, I don't find any error message or anomaly related to this step. @yuankailiu and @olliestephenson also had the same issue. @CunrenLiang any idea why it happened?

(base) honshu: SaltonSeaSenDT173/ion>$ ll 20151223_20160304/lower/merged
total 28M
drwxr-xr-x 2 zyunjun    5 Jun  8 18:15 ./
drwxr-xr-x 4 zyunjun    4 Jun  8 18:15 ../
-rw-r--r-- 1 zyunjun  31M Jun  8 18:21 fine.int
-rw-r--r-- 1 zyunjun  21K Jun  8 18:15 fine.int.full.vrt
-rw-r--r-- 1 zyunjun 4.0K Jun  8 18:15 fine.int.full.xml
(base) honshu:/marmot-nobak/zyunjun/offset4motion/SaltonSeaSenDT173/ion>$ ll 20200617_20200711/lower/merged/
total 88M
drwxr-xr-x 2 zyunjun   22 Jun  9 04:40 ./
drwxr-xr-x 4 zyunjun    4 Jun  8 23:22 ../
-rw-r--r-- 1 zyunjun  31M Jun  8 23:33 fine.cor
-rw-r--r-- 1 zyunjun  694 Jun  8 23:33 fine.cor.vrt
-rw-r--r-- 1 zyunjun 3.9K Jun  8 23:33 fine.cor.xml
-rw-r--r-- 1 zyunjun  31M Jun  8 23:27 fine.int
-rw-r--r-- 1 zyunjun  21K Jun  8 23:22 fine.int.full.vrt
-rw-r--r-- 1 zyunjun 4.0K Jun  8 23:22 fine.int.full.xml
-rw-r--r-- 1 zyunjun  469 Jun  8 23:27 fine.int.vrt
-rw-r--r-- 1 zyunjun 4.2K Jun  8 23:27 fine.int.xml
-rw-r--r-- 1 zyunjun 1.3M Jun  9 04:40 fine_look.cor
-rw-r--r-- 1 zyunjun  700 Jun  9 04:40 fine_look.cor.vrt
-rw-r--r-- 1 zyunjun 3.9K Jun  9 04:40 fine_look.cor.xml
-rw-r--r-- 1 zyunjun 1.3M Jun  9 04:40 fine_look.unw
-rw-r--r-- 1 zyunjun  700 Jun  9 04:40 fine_look.unw.vrt
-rw-r--r-- 1 zyunjun 4.2K Jun  9 04:40 fine_look.unw.xml
-rw-r--r-- 1 zyunjun  31M Jun  9 03:33 fine.unw
-rw-r--r-- 1 zyunjun 3.9M Jun  9 03:33 fine.unw.conncomp
-rw-r--r-- 1 zyunjun  383 Jun  9 03:33 fine.unw.conncomp.vrt
-rw-r--r-- 1 zyunjun 3.6K Jun  9 03:33 fine.unw.conncomp.xml
-rw-r--r-- 1 zyunjun  694 Jun  9 03:33 fine.unw.vrt
-rw-r--r-- 1 zyunjun 3.8K Jun  9 03:33 fine.unw.xml

[Fei0906]

There is a bug in stackSentinel.py in line 544, where the statement of "cnt += 1" is not under the if statement due to the indentation issue. This bug causes the network is not fully connected, making the invertIon step impossible. However, I am not sure if this bug has any causal relationship with the error you mentioned here.

[yuankailiu]

@Fei0906 , you are correct. the "cnt+=1" is at a weird location. I have a commit that fixes this issue, which can be a PR. This bug I think will only affect the overall pairing. All the run files should still have consistent pairs.

@yunjunz , I am still not sure what it is now. Those missing pair problems for me occurred when I was trying to pair S1A and S1B. I have not found out why. But I will look further into that.

[yunjunz]

Thank you @Fei0906 and @yuankailiu for pointing out the indentation bug, please issue the PR to fix it when you got a chance please.