How to set ISCE stackSentinel.py parameters in order to run MiaplPy?

[leeyhgeo]

Hello,

I used to analyze Sentinel1 images with MintPy and I found that MiaplPy seems to need different input. However, I'm confused about how to set stackSentinel.py parameters so that I can get coregistered SLC data in full resolution. Is there any document/discussion that could be referenced? (I can hardly find one so far)

Thank you

[mirzaees]

Hi @geoglee MiaplPy can read data from 'interferogram' and 'slc' workflow of stackSentinel.py

If your current directory is the project directory, then setting the autoPath to yes would read all data automatically. You might need to rename your template to reflect your project, for example if your project directory name is 'testProject' then rename your template to testProject.template:

./testProject/testProject.template

https://github.com/insarlab/MiaplPy/blob/3771804c0fc168df6c9a9114ab1fd74a5e19ee70/miaplpy/defaults/miaplpyApp.cfg#L22

Also you can manually set path to data as bellow in your template file:

https://github.com/insarlab/MiaplPy/blob/3771804c0fc168df6c9a9114ab1fd74a5e19ee70/miaplpy/defaults/auto_path.py#L21-L38

[leeyhgeo]

Thank you so much @mirzaees ! I've tried your suggestion but I ran into some path problems. I hope I could solve it soon. At the same time, I'm using stackStripMap.py to process ERS and Envisat data. Is it possible as well to run products from stackStripMap.py in MiaplPy?

[mirzaees]

you are very welcome @geoglee I may be able to help you if you send the error you get and your template

Regarding stripmap, yes, I have run it for TerraSAR data before and it worked however I have never tested ERS or Envisat, let me know if you encounter a problem

[Jesse-kearse]

Hello

I have been using MintPy and MiaplPy, and have managed to get good results. Recently, I have had an issue when loading an SLC stack.

The SLC stack was generated with the following parameters: stackSentinel.py -s /insar/raw/track_08/ -d /insar/raw/dem/dem.wgs84 -b '-38.16 -37.53 176.07 177.73' -n '1 2' -m 20180112 -a ./aux/ -o ./orbits -C NESD -W slc

I ran the MiaplPy software with the suggested command "miaplpyApp.py kawerau.cfg --runfiles --dir ."

the prepc_slc_isce.py application crashes, and I am unsure why. I thought it may be due to 1) "miaplpy.subset.lalo" setting, so I tried with various extents and with no subset, and it was the same result 2) "miaplpy.load.startDate" or "maiplpy.load.endDate" so I tried with and without dates and the result was the same.

The only difference I could think of between this SLC stack and other SLC stack that worked fine with miaplpy, is co-registration method: "NESD" was used here, and I had used "geometry" in the past with the SLC stack that were loaded into Miaplpy without issue. Perhaps the the choice of number of swaths could be a factor? I am stuck as to the issue.

Below is the report of the run, is there anything else I could provide to help diagnosing the issue?

Thanks for any help

--RUN-at-2023-09-25 11:35:24.796217-- Current directory: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy Run routine processing with arg_parser.py on steps: ['load_data', 'phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction'] Remaining steps: ['phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction'] Project name: kawerau

20230925:113524 miaplpyApp.py kawerau.cfg --runfiles --dir . Project name: kawerau Go to work directory: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy copy default template file /home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/defaults/smallbaselineApp.cfg to work directory read custom template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/kawerau.cfg update default template based on input custom template No new option value found, skip updating /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/smallbaselineApp.cfg copy kawerau.cfg to inputs directory for backup. copy smallbaselineApp.cfg to inputs directory for backup. copy kawerau.cfg to pic directory for backup. copy smallbaselineApp.cfg to pic directory for backup. read default template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/smallbaselineApp.cfg copy default template file /home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/objects/../defaults/miaplpyApp.cfg to work directory update default template based on input custom template miaplpy.multiprocessing.numProcessor: auto --> 40 miaplpy.load.processor: auto --> isce miaplpy.load.autoPath: auto --> yes miaplpy.load.slcFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/SLC//.slc.full miaplpy.load.startDate: auto --> 20180112 miaplpy.load.endDate: auto --> 20200114 miaplpy.load.metaFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW.xml miaplpy.load.baselineDir: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/baselines miaplpy.load.demFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/hgt.rdr.full miaplpy.load.lookupYFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/lat.rdr.full miaplpy.load.lookupXFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/lon.rdr.full miaplpy.load.incAngleFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/los.rdr.full miaplpy.load.azAngleFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/los.rdr.full miaplpy.load.shadowMaskFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/shadowMask.rdr.full miaplpy.load.waterMaskFile: auto --> None miaplpy.load.unwFile: auto --> ./inverted/interferograms_single_reference//fine.unw miaplpy.load.corFile: auto --> ./inverted/interferograms_single_reference//fine.cor miaplpy.load.connCompFile: auto --> ./inverted/interferograms_single_reference//.unw.conncomp miaplpy.subset.lalo: auto --> -38.117:-37.995,176.651:176.913 mintpy.load.processor: auto --> isce copy miaplpyApp.cfg to inputs directory for backup. read default template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/miaplpyApp.cfg SAR platform/sensor : None processor: isce check auto path setting for Univ of Miami users for processor: isce

prepare metadata files for isce products prep_slc_isce.py -s /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/SLC -f *.slc.full -m /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW1.xml -b /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/baselines -g /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference --force extract metadata from ISCE/topsStack xml file: /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW1.xml extract metadata from geometry files: ['hgt.rdr.full', 'lat.rdr.full', 'lon.rdr.full', 'los.rdr.full'] extract metadata from geometry files: ['hgt.rdr', 'lat.rdr', 'lon.rdr', 'los.rdr'] Traceback (most recent call last): File "/home/kearsejess/conda/envs/mintpy_new/bin/prep_slc_isce.py", line 8, in sys.exit(main()) ^^^^^^ File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/prep_slc_isce.py", line 362, in main metadata = extract_isce_metadata(inps.metaFile, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/prep_slc_isce.py", line 145, in extract_isce_metadata metadata_multi_looked = isce_utils.extract_geometry_metadata(geom_dir, metadata_multi_looked) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/isce_utils.py", line 559, in extract_geometry_metadata data = readfile.read(geom_file, box=box)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 310, in read data, atr = read_binary_file(fname, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 679, in read_binary_file data = read_binary( ^^^^^^^^^^^^ File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 1914, in read_binary data = np.fromfile(fname, dtype=data_type, count=count).reshape(-1, widthnum_band) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: cannot reshape array of size 15521796 into shape (43903)

[Jesse-kearse]

P.S. I am running the latest version of MiaplPy V0.2.1, which was downloaded ~20092023

[mirzaees]

Hi @Jesse-kearse Looks like your program crashes when reading geometry files. One thing that comes to my mind is the *.rsc files, If you have run mintpy before this, it has created *.rsc metadata in the geometry directory which are associated with multilooked files. now you are reading the full res and miaplpy tries to create *.rsc for full res but it first checks whether they exist in the directory. So please delete them (all *.rsc files in merged/geom_reference directory and merged/SLC/dates/* directory) and try again. let me know if that solves the issue

[Jesse-kearse]

Hi @mirzaees , yes I had run mintpy on the same SLC stack prior to miaplpy. Thanks for your advice! I will try your suggestion and let you know how it goes.

Cheers

[Jesse-kearse]

hi @mirzaees , Actually, I had not run mintpy on this particular SLC stack, and there were no .rsc files in the /merged/geom_reference/ dir or the /merged/SLC/*/ directory. Any other ideas?

Thanks again for your help.

[mirzaees]

Hi @Jesse-kearse, I need a subset from your data as a test to run it and see if I can reproduce your error. Can you provide me your merged folder with geom-reference, and 2 dates in SLC and reference+baseline folders in a zipfile somewhere?

[gjustin40]

I also have a similar issue. @mirzaees I created a stack using ISCE2, and ultimately, the problem is that the slc.full data is missing.

miaplpy.load.slcFile        = ../merged/SLC/*/*.slc.full  #[path2slc_file]

After running ISCE2 command, the slc.full file was not generated. The ISCE2 command I executed is as follows:

dem.py -a stitch -b {bbox lat/lon} -r -s 1 -c
stackSentinel.py -s SLC/ -d DEM/*.dem.wgs84 -a AuxDir/ -o Orbits/ -b {bbox lat/lon} -c 2
# run all run_files

I successfully used MintPy with the files generated by above command.

I know there are a lot of example of command to make run_files in ISCE, What command should I execute in ISCE2 to generate slc.full files?

Thanks

[Jesse-kearse]

Hi @mirzaees

Thank you for offering to take a look.

I have created the a zip archive "SLC_zip.zip" that contains the files you requested. You can download with the following command:

rsync -av --progress rsync://kearsejess@core.geo.vuw.ac.nz/kearsejess/

password: gJbmvnmtUU

Thanks again.

[mirzaees]

Hi @Jesse-kearse Thanks for providing your data. which workflow of stackSentinel.py have you used? if that is interferogram or slc, then it should work the way it is without any touching but from your data I realize you may have done some separate/post processing or the merging step was not complete. that is because your multilooked version of geometry files are not compatible with their shape in VRT. Also if you display them with view.py of mintpy, you get a similar error. Anyways I removed that part of the code that relies on multilooked version and should not be a problem any more, just reinstall with:

cd MiaplPy; git pull
python -m pip install .

one more issue is about the baselines you provided which seems to be a raster file for each date. however, miaplpy assumes text files in the following format if your reference date is 20180112:

20180112_20180301/20180112_20180301.txt
20180112_20180313/20180112_20180313.txt

I don't know why your baselines are like this but stackSentinel.py should create those text files. You will still get error if your baselines are not in that format

[mirzaees]

Hi @gjustin40 As long as the full res VRT files exist and you have not deleted the other folders in the project, you are fine and it should be able to read them. However, as I mentioned above, if you have run mintpy before miaplpy, clean all .rsc metadata and run miaplpy again

[Jesse-kearse]

Thanks for taking the time to look at the files, @mirzaees , I really appreciate your help.

The workflow I used was SLC: "stackSentinel.py -s /insar/raw/track_08/ -d /insar/raw/dem/dem.wgs84 -b '-38.16 -37.53 176.07 177.73' -n '1 2' -m 20180112 -a ./aux/ -o ./orbits -C NESD -W slc"

I realise now that the post processing that I had done with StaMPS (which includes a multi-looking step) was probably the reason for the modified geometry files. But with your recent update this will no longer be an issue! thanks, I will git pull and re-install.

The baseline files I sent were from ./merged/baselines/ dir. Sorry I misinterpreted which baseline information you wanted. The files in ./baselines/ dir are in the correct format: 20180112_20180301/20180112_20180301.txt, so that should be fine.

I will let you know how it goes.

Thanks again.

[Jesse-kearse]

Following git pull and re-installation it was a successful completion of "miaplpyApp file.cfg --runfiles --dir ."

Thanks @mirzaees

[mirzaees]

You are very welcome @Jesse-kearse I will close this issue

[leeyhgeo]

Hi @geoglee MiaplPy can read data from 'interferogram' and 'slc' workflow of stackSentinel.py

If your current directory is the project directory, then setting the autoPath to yes would read all data automatically. You might need to rename your template to reflect your project, for example if your project directory name is 'testProject' then rename your template to testProject.template:

./testProject/testProject.template

https://github.com/insarlab/MiaplPy/blob/3771804c0fc168df6c9a9114ab1fd74a5e19ee70/miaplpy/defaults/miaplpyApp.cfg#L22

Also you can manually set path to data as bellow in your template file:

https://github.com/insarlab/MiaplPy/blob/3771804c0fc168df6c9a9114ab1fd74a5e19ee70/miaplpy/defaults/auto_path.py#L21-L38

@mirzaees Sorry for the really late reply.

I run my stackSentinel.py with the following parameter stackSentinel.py -s ~/slc -o ~/orbits -a ~/S1_auxiliary_files -d ~/dem/dem.wgs84 -c 3 -z 3 -r 15 -b '22.3 24.4 119.9 120.6' and this is how it looks like under the folder ${PROJECT_DIR}/merged/SLC/*

I cannot find any files end with *.slc.full, and so as files under the geom_reference folder as below

I wonder if the problem arises from my multilooking settings of -z 3 -r 15 above, or maybe other issues.

Thanks!

[mirzaees]

@geoglee I don't see any problem with your isce outputs, as long as the vrt files exist in your folder, miaplpy will read the data. you need to set up your template similar to the example above. I need to see your error to be able to help you, can you run and send me your error message?

[Jesse-kearse]

@geoglee

If you only have the "merged" directory and sub directories, you may be missing the files that the .slc.full.vrt files are referencing. In my workflow, .slc.full.vrt files point to burst.slc.vrt files in the "coreg_secondarys" directory, one above "merged". The raw data is contained in the burst.slc files within the same directory. I ran into this issue by mistake when I deleted a whole bunch of files (to gain extra disc space) thinking I only needed the merged// files. You can check by opening the *.slc.full.vrt files and checking the path in

[Jesse-kearse]

...checking the path in SourceFilename relativeToVRT

[leeyhgeo]

@mirzaees @Jesse-kearse

Thank you a lot for helping me! For the path problem, awkwardly, I rechecked the template and just found that I didn't turn on the auto path function since I thought the default setting (auto) was yes till I noticed the comment behind miaplpy.load.autoPath. As I turn on the auto path function, Miaplpy seems to work right now (though it's still running). Hope it can operate normally till the final step. I'll report the outcome later.

Another question about the stackSentinel.py settings. I'm not sure whether multilooking (-z, -r) and num_connections (-c) will affect the Miaplpy processing results. Any ideas?

Besides, I already put the keyword "Sen" into the file name of my template, but the message still outputs SAR platform/sensor : None. How to set it properly?