concatenating_chunks.py suggestions

[falkamelung]

Hi @ranneylxr: Here some more suggestions

• We don't need a mimtpy folder, right? (that makes only sense for concatenate_tracks.py. ) When calling concatenate_chunks.py --chunks ChamanChunk3{1,2}SenAT144 --project ChamanBigSenAT144 it should go into ChamanBigSenAT144/mintpy. And it should say --dir mintpy in the example messages.
• We better leave the chunk folders untouched and write all files (including temporary files) into ChamanBigSenAT144.
• We could have

  ChamanBigSenAT144/mintpy/ChamanChunk31SenAT144/geo_velocity_track.h5
  ChamanBigSenAT144/mintpy/ChamanChunk31SenAT144/geo_temporalCoherence_track.h5
  ChamanBigSenAT144/mintpy/ChamanChunk32SenAT144/geo_velocity_track.h5
  ChamanBigSenAT144/mintpy/ChamanChunk32SenAT144/geo_temporalCoherence_track.h5

• --all or --HDFEOS5 to create all. This also should be the default.
• --clean option (or --remove_temporary_files) to remove all temporary files
• The files produced could be following MintPy convention (instead of *_track.h5):

  ChamanBigSenAT144/mintpy/geo/geo_velocity.h5
  ChamanBigSenAT144/mintpy/geo/geo_temporalCoherence.h5
  etc

  Then we could run smallbaselineApp.py --plot and save_kmz.py.

• create the HDFEOS5 files using save_hdfeos5.py ?
• the S1 file should be named similar to what it was (S1_IW123_071_0105_0110_20141010_XXXXXXXX.he5). Currently it is S1_20141010_20210705_track.he5. You can just use the startFrame and endFrame from the S1 files. For the startDate and endDate we need a convention. If the startDate get changed use the new one. If the endDate is XXXXXXXX keep it. If the endDate is a date and it got changed use the new one.
• in the S1 file mask and shadowMask are missing. mask is particularly important as tsview.py needs it.
• use grayscale for avgSpatialCoherence.png

Minor comments:

• Instead of Delete velocity folder from /scratch1/05861/tg851601/ChamanChunk31SenAT144/mintpy say Deleting /scratch1/05861/tg851601/ChamanChunk31SenAT144/mintpy/velocity

• it is HDFEOS5 (with a 5)

• It says timeseries2velocity.py S1_IW123_144_0102_0107_20141027_XXXXXXXX.he5. Better to use and display full paths.

• dry run option: Currently it says:

  The removed date for project /scratch1/05861/tg851601/ChamanChunk25SenAT144/mintpy:
  [b'20170426']
  The removed date for project /scratch1/05861/tg851601/ChamanChunk26SenAT144/mintpy:
  [b'20151022', b'20170426', b'20150928']
  The removed date for project /scratch1/05861/tg851601/ChamanChunk27SenAT144/mintpy:
  [b'20151022', b'20150507', b'20170426', b'20150928']
  The removed date for project /scratch1/05861/tg851601/ChamanChunk28SenAT144/mintpy:
  [b'20151022', b'20150507', b'20170426', b'20150928']
  The removed date for project /scratch1/05861/tg851601/ChamanChunk29SenAT144/mintpy:
  [b'20151022', b'20150507', b'20170426', b'20150928']
  The removed date for project /scratch1/05861/tg851601/ChamanChunk30SenAT144/mintpy:
  [b'20220211', b'20150928', b'20151022', b'20170426', b'20150507']
  The removed date for project /scratch1/05861/tg851601/ChamanChunk31SenAT144/mintpy:
  [b'20220211', b'20150928', b'20151022', b'20170426', b'20150507']

  Improve formatting (remove the b) and sort by time. Instead of "The removed date for project " say "The removed dates for chunk:" (without mintpy)

• When a concatenated timeseries is created, write the same info in a file (e.g. timeseries_concatenation.txt.

• in the --help message explain a bit more. Why are *.tiff generated (I see only temporalCoherence.tiff, avgSpatialCoherence.tiff). Also explain the order of processing for HDFEOS.

• in the --help message explain what happens if a chunk is missing (e.g. could not unwrap. It seems to me that there is a white space, which is good. How is offset calculated if there is no overlap area?

• A white line before processing each chunk. Or we follow the MintPy example which has some markers before a new python script is called:

**** step - geocode ****

geocode.py /scratch1/05861/tg851601/unittestGalapagosSenDT128/mintpy/inputs/geometryRadar.h5 temporalCoherence.h5 avgSpatialCoh.h5 /scratch1/05861/tg851601/unittestGalapagosSenDT128/mintpy/timeseries_SET_ERA5_demErr.h5 velocity.h5 -l /scratch1/05861/tg851601/u nittestGalapagosSenDT128/mintpy/inputs/geometryRadar.h5 -t

### Typos/spelling errors:
- exsited
- poitn

### Questions:
-It says `6. The concatenated data are masked data based on the mask of each chunk`.  If I have a S1 file with concatenated data I can use a different tempCoherence threshold, right?

[ranneylxr]

Hi @falkamelung, Now all comments are solved besides the followings:

1. --all or --HDFEOS5 to create all. This also should be the default. Answer: I perfer to set the HDFEOS5 for the --datatype option. Or we could make the HDFEOS5 as default for the --datatype option?
2. create the HDFEOS5 files using save_hdfeos5.py ? Answer: I tried to generate HDFEOS5 files based on the concatenated velocity, etc files. But there are errorrs message saying that we need a template file... Therefore, I decided to generate HDFEOS5 file in the concatenate_chunks.pyscript.
3. use grayscale for avgSpatialCoherence.png Answer: The default color used byview.py for the avgSpatialCoherence.py is not gray... Does that matter?
4. When a concatenated timeseries is created, write the same info in a file (e.g. timeseries_concatenation.txt.) Answer: I am not fully understand it.... Do you mean we need to save the info shown in the screen into a txt file?
5. -It says 6. The concatenated data are masked data based on the mask of each chunk. If I have a S1 file with concatenated data I can use a different tempCoherence threshold, right? Answer: The temporalCoherence extracted from the S1 file of each chunks is masked by the mask info of the S1 file. So it seems that a threshold smaller than 0.7 has no meaning for the concatenated file?

[falkamelung]

Hi @ranneylxr These are all minor issues:

3. Spatial coherence is always shown in grey scale. View.py has a different default as it is mainly used for velocity. If adding -colorscale gray -v 0 1 or similar addresses this lets just do it.
4. Yes. Just keep something in the folder. It is currently not tracked which files were concatenated.

You said somewhere smallbaselineApp.py --plot does not work (I did not try). Can you elaborate why? Some files missing? Would it work if plot.py just checks for the existence of the files? The other alternative is to modify the old plot*.sh script for plotting (I can do that)