geodesymiami / mimtpy

MIami Multi Track tools in Python (or MIami Modelling Tools)
10 stars 6 forks source link

Multitrack directory structure #1

Open falkamelung opened 5 years ago

falkamelung commented 5 years ago

Lets agree here about the multitrack directory structure.

Here is Yunjun's directory structure:

[yunjunz:~/insarlab/Kirishima] yunjunz% vi $TE/KirishimaAlosAT424F620_630.template 

yunjunz:~] yunjunz% cd $KIRISHIMA_DIR/
[yunjunz:~/insarlab/Kirishima] yunjunz% ll
drwxr-xr-x@ 14 yunjunz  staff   476B May 13 23:35 ../
drwxr-xr-x   9 yunjunz  staff   306B Jul 18 21:49 Alos2DT23F2970/
drwxr-xr-x  18 yunjunz  staff   612B May 20 22:27 AlosAT424F620_630/
drwxr-xr-x   5 yunjunz  staff   170B Apr 15 18:09 AlosDT72F2980/
drwxr-xr-x  13 yunjunz  staff   442B Jun 18 16:23 AlosDT73F2970_2980/
drwxr-xr-x   6 yunjunz  staff   204B Jul  5 17:35 Asc_desc/
drwxr-xr-x  20 yunjunz  staff   680B Aug 14 17:41 DEM/
drwxr-xr-x   7 yunjunz  staff   238B Apr 14 19:46 Data/
drwxr-xr-x  35 yunjunz  staff   1.2K Jul  4 00:37 Doc/
drwxr-xr-x  17 yunjunz  staff   578B Apr 22 12:50 GPS/
drwxr-xr-x  10 yunjunz  staff   340B Jun 29 17:57 KirishimaAlos2DT23/
drwxr-xr-x  10 yunjunz  staff   340B Jul 20 20:29 Model/
drwxr-xr-x   9 yunjunz  staff   306B Aug 16 22:35 Seismic/
drwxr-xr-x   5 yunjunz  staff   170B Aug  4 18:24 ShinmoeAlosAT424/
drwxr-xr-x   5 yunjunz  staff   170B Aug  4 18:24 ShinmoeAlosDT73/
drwxr-xr-x  12 yunjunz  staff   408B Apr  9 14:50 Tephra/
[yunjunz:~/insarlab/Kirishima] yunjunz% ll Model/
drwxr-xr-x   6 yunjunz  staff   204B Jul  1 21:42 GeodMod/
drwxr-xr-x  11 yunjunz  staff   374B Jul 20 02:22 Shinmoe_2008/
-rw-r--r--@  1 yunjunz  staff   8.5K Jul 20 22:35 Shinmoe_2008.inp
drwxr-xr-x   6 yunjunz  staff   204B Jun 30 20:47 Shinmoe_2016/
-rw-r--r--@  1 yunjunz  staff   6.3K Jul  1 00:53 Shinmoe_2016.inp
drwxrwxrwx  23 yunjunz  staff   782B Jul  9 14:47 dMODELS/
drwxr-xr-x  15 yunjunz  staff   510B Jul 21 18:54 data/
[yunjunz:~/insarlab/Kirishima] yunjunz% ll Model/data/
-rw-r--r--   1 yunjunz  staff   830K Jun 17 13:12 Alos2DT23_20150223_20161031.cor
-rw-r--r--   1 yunjunz  staff   2.0K Jun 17 13:12 Alos2DT23_20150223_20161031.cor.rsc
-rw-r--r--   1 yunjunz  staff   5.2M Jun 30 20:21 Alos2DT23_20150223_20161031.mat
-rw-r--r--   1 yunjunz  staff   936K Jun 30 20:21 Alos2DT23_20150223_20161031.png
-rw-r--r--   1 yunjunz  staff   830K Jun 17 13:11 Alos2DT23_20150223_20161031_msk.unw
-rw-r--r--   1 yunjunz  staff   2.1K Jun 17 13:11 Alos2DT23_20150223_20161031_msk.unw.rsc
-rw-r--r--   1 yunjunz  staff   1.1M Jul 20 16:42 ShinmoeAlosAT424_20080929_20100520.mat
-rw-r--r--   1 yunjunz  staff   413K Jul 20 16:42 ShinmoeAlosAT424_20080929_20100520.png
-rw-r--r--   1 yunjunz  staff   1.0M Jul 20 16:42 ShinmoeAlosDT73_20081012_20100302.mat
-rw-r--r--   1 yunjunz  staff   433K Jul 20 16:42 ShinmoeAlosDT73_20081012_20100302.png
-rw-r--r--   1 yunjunz  staff   397K Jul 21 18:54 ShinmoeHz.h5
-rw-r--r--   1 yunjunz  staff   397K Jul 21 18:54 ShinmoeUp.h5

[yunjunz:~/insarlab/Kirishima] yunjunz% ll ShinmoeAlosAT424/mintpy/
[yunjunz:~/insarlab/Kirishima] yunjunz% ll AlosAT424F620_630/
drwxr-xr-x    8 yunjunz  staff   272B Jul  6 01:04 20101120_20110220/
drwxrws---   18 yunjunz  staff   612B Apr  1 16:59 DEM/
drwxrws---  227 yunjunz  staff   7.5K Apr 15 17:36 Igrams/
drwxrws---   30 yunjunz  staff   1.0K Apr 15 17:36 baselines/
drwxrws---  566 yunjunz  staff    19K May 14 12:36 configs/
drwxr-xr-x   92 yunjunz  staff   3.1K Apr  7 19:14 gacos/
drwxrws---   41 yunjunz  staff   1.4K Apr 15 22:28 geom_master/
-rw-r--r--@   1 yunjunz  staff   1.9K Apr  5 17:36 master.xml
drwxr-s---    6 yunjunz  staff   204B Apr 15 22:28 masterShelve/
drwxr-xr-x   10 yunjunz  staff   340B Aug 10 22:06 mintpy/
-rw-rw----    1 yunjunz  staff    82K Apr 14 23:30 pairs.pdf
drwxrws---   29 yunjunz  staff   986B Apr 15 17:36 run_files/
-rw-rw----    1 yunjunz  staff   5.5K Apr 14 23:27 run_unPackALOS

[yunjunz:~/insarlab/Kirishima] yunjunz% ll AlosAT424F620_630/
falkamelung commented 5 years ago

Current helps:

//login3/nethome/famelung/test/development/rsmas_insar/sources/MimtPy[1013] save_geodmod.py --help
usage: save_geodmod.py [-h] [-t TEMPLATEFILE] [-ds [DATASET]] [-dt [DATATYPE]]
                       [-b S N W E] [-y LATSTEP] [-x LONSTEP] [-s [STARTDATE]]
                       [-e [ENDDATE]] [-outdir OUTDIR]
                       [file]

Prepare data for Geodmod software

positional arguments:
  file                  ascending and descending timeseries files

optional arguments:
  -h, --help            show this help message and exit
  -t TEMPLATEFILE, --template TEMPLATEFILE
                        Template file with geocoding options.
  -ds [DATASET], --dataset [DATASET]
                        name of dataset.Seperating by ','. 
  -dt [DATATYPE], --datatype [DATATYPE]
                        clarify the type of data.[velocity, ifgramStack, timeseries].Only used in template file
  -b S N W E, --bbox S N W E
                        Bounding box of area to be geocoded.
                        Include the uppler left corner of the first pixel    and the lower right corner of the last pixel
  -y LATSTEP, --latstep LATSTEP
                        output pixel size in degree in latitude.
  -x LONSTEP, --lonstep LONSTEP
                        output pixel size in degree in longitude.
  -s [STARTDATE], --startDate [STARTDATE]
                        date1 of timeseires to be converted.The default is the StartDate
  -e [ENDDATE], --endDate [ENDDATE]
                        date2 of timeseries to be converted.The default is the EndDate
  -outdir OUTDIR, --outdir OUTDIR
                        output directory

example:
  for singletrack:
  Note: startDate, endDate and outdir have default values, any of them can be not given:
    startDate default value = atr['START_DATE']
    endDate default value = atr['END_DATE']
    outdir default value = '$MODELDIR/project/Sen**/geodmod_startDate_endDate/'
  save_geodmod.py timeseries_ECMWF_demErr.h5 -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001  
  save_geodmod.py ifgramStack.h5  -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001 -s 20171117 -e 20171129 -outdir $MODELDIR/Darbandikhan/SenAT73/
  save_geodmod.py velocity.h5  -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001 -s 20171117 -e 20171129 -outdir $MODELDIR/Darbandikhan/SenAT73/
  save_geodmod.py S1_IW23_026_0108_0113_20171117_XXXXXXXX.he5 -s 20171128 -e 20181210 
  for multitrack:
  Note: startDate, endDate and DataSet can be not given in template:
  save_geodmod.py -t $MIMTFILES/Darbandikhan.txt

And for save_gbis.py:

./save_gbis.py --help
usage: save_gbis.py [-h] [-t TEMPLATEFILE] [--model-software MODEL_SOFTWARE]
                    [-ds [DATASET]] [-dt [DATATYPE]] [-b S N W E] [-y LATSTEP]
                    [-x LONSTEP] [-s [STARTDATE]] [-e [ENDDATE]]
                    [-m MASK_FILE] [--ref-lalo [REF_LALO]] [-outdir OUTDIR]
                    [file]

Prepare data for GBIS software

positional arguments:
  file                  ascending or descending files

optional arguments:
  -h, --help            show this help message and exit
  -t TEMPLATEFILE, --template TEMPLATEFILE
                        Template file with geocoding options.
  --model-software MODEL_SOFTWARE
                        Goephysical model's name.
  -ds [DATASET], --dataset [DATASET]
                        name of dataset.Seperating by ','. 
  -dt [DATATYPE], --datatype [DATATYPE]
                        clarify the type of data.[velocity, ifgramStack, timeseries, S1*.he5].Only used in template file
  -b S N W E, --bbox S N W E
                        Bounding box of area to be geocoded.
                        Include the uppler left corner of the first pixel    and the lower right corner of the last pixel
  -y LATSTEP, --latstep LATSTEP
                        output pixel size in degree in latitude.
  -x LONSTEP, --lonstep LONSTEP
                        output pixel size in degree in longitude.
  -s [STARTDATE], --startDate [STARTDATE]
                        date1 of timeseires to be converted.The default is the StartDate
  -e [ENDDATE], --endDate [ENDDATE]
                        date2 of timeseries to be converted.The default is the EndDate
  -m MASK_FILE, --mask MASK_FILE
                        mask file.
  --ref-lalo [REF_LALO]
                        custom reference pixel in lat/lon
  -outdir OUTDIR, --outdir OUTDIR
                        output directory

example:
  for singletrack:
  Note: startDate, endDate and outdir have default values, any of them can be not given:
    startDate default value = atr['START_DATE']
    endDate default value = atr['END_DATE']
    outdir default value = '$MODELDIR/project/Sen**/gbis_startDate_endDate/'
  save_gbis.py timeseries_ECMWF_demErr.h5 --model-software gbis -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001  
  save_gbis.py ifgramStack.h5 --model-software gbis -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001 -s 20171117 -e 20171129 -outdir $MODELDIR/Darbandikhan/SenAT73/
  save_gbis.py velocity.h5 --model-software gbis -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001 -s 20171117 -e 20171129 -outdir $MODELDIR/Darbandikhan/SenAT73/
  save_gbis.py S1_IW23_026_0108_0113_20171117_XXXXXXXX.he5 --model-software gbis -s 20171128 -e 20181210 
  for multitrack:
  Note: startDate, endDate and DataSet can be not given in template:
  save_gbis.py -t $MIMTFILES/Darbandikhan.txt
falkamelung commented 5 years ago

The geo_* files that save_geodmod produces have a different posting then what we have in the geo directory. Shall we place them into a tmp_geo directory to avoid confusion?

ranneylxr commented 5 years ago

I design the structure of directory as following: Firstly, the template files for mimtpy are stored in MIMTFILES; the generated results are stored in MODELDIR; Data that mimtpy used(generated by mintpy) are stored in SCRATCHDIR.

//centos7/data2/rsmasdata/lxr[1005] ll
drwxrwxr-x  2 lxr lxr       38 Aug 17 11:18 MIMTFILES
drwxrwsr-x  3 lxr famelung  34 Aug 21 09:17 MODELDIR
drwxrwxr-x 13 lxr lxr      334 Aug 20 15:28 SCRATCHDIR
drwxrwxr-x  4 lxr lxr       61 Jun 21 16:13 TESTDATA_ISCE

Secondly, I want to design the directory structure of MODELDIR like

$MODELDIR/project/SenAT(DT)***/gbis_startdate_enddate
$MODELDIR/project/SenAT(DT)***/geodmod_startdate_enddate
$MODELDIR/project/horzvert/startdate_enddate

Example:

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan[1009] ll
drwxrwsr-x 4 lxr famelung  68 Aug 20 20:46 horzvert
drwxrwsr-x 6 lxr famelung 149 Aug 19 20:42 SenAT72
drwxrwsr-x 9 lxr famelung 254 Aug 19 20:49 SenAT73
drwxrwsr-x 4 lxr famelung  81 Aug 19 17:14 SenDT79

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/SenAT72[1011]
gbis_20171117_20171129  
gbis_20171123_20171223
gbis_20171117_20171223 
geodmod_20171117_20171223

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/SenAT72/gbis_20171117_20171129/
DarbandikhanSenAT72_20171117_20171129.mat
geo_20171117_20171129.unw
geo_20171117_20171129.unw.rsc
//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/SenAT72/geodmod_20171117_20171223/
geo_20171117_20171223.cor
geo_20171117_20171223.cor.rsc
geo_20171117_20171223.unw
geo_20171117_20171223.unw.rsc
srtm.dem
srtm.dem.jpeg
srtm.dem.jpeg.rsc
srtm.dem.rsc
//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/horzvert/
20171117_20171205
0171117_20190423

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/horzvert/20171117_20190423/
geo_20171117_20190423.unw
geo_20171117_20190423.unw.rsc
geo_20171118_20190430.unw
geo_20171118_20190430.unw.rsc
hor.h5
hz.h5
up.h5
vel_horizontal.h5
vel_vertical.h5
ver.h5
ranneylxr commented 5 years ago

For the geofile in geod , gbis* and horzvert folders, do you mean we put the geofile into geod/tmp_geo/ folder?

falkamelung commented 5 years ago

No. We don’t need them, right? I mean /mintpy/tmp_geo

ranneylxr commented 5 years ago

Yes, we don't need geo_***.h5 files. How about just delete them?

falkamelung commented 5 years ago

Yes. Write them in the tmp_* and then delete.

On Aug 20, 2019, at 11:37 PM, ranneylxr notifications@github.com<mailto:notifications@github.com> wrote:

Yes, we don't need geo_***.h5 files. How about just delete them?

ranneylxr commented 5 years ago

OK! I will change the code

falkamelung commented 5 years ago

@ranneylxr : If we follow Yunjun's suggestion and have save_gbis.py and save_geodmod.py only do very simple things. then we could have a save_for_modelling.py --modelSoftware gbis that takes care of the multitrack issues. All the functions such as track_date could be in mimtpy/utils/multitrack_utilities.py

By the way, some of the functions don't fave self-explanatory names. find_date --> find_start_end_date.py, track_date.py -->find_nearest_date.py

falkamelung commented 5 years ago

@ranneylxr , I did some minor modifications as you have seen while I was debugging. Here a few more comments on the code:

ranneylxr commented 5 years ago

OK. I will try to change them.

ranneylxr commented 5 years ago

The new usage of save_geodmod.py and save_gbis_mimt.py is:

  1. save_for_modelling.py process multitrack information. It support geodmod software and gbis software now.
  2. save_geodmod.py and save_gbis both support four kind of data types(HDFEOS/ifgramStack/timeseries/velocity). Both of them can only be used for single track. And you have to call this command under $SCRATCHDIR/project/mintpy/ directory
  3. The default outdir is $SCRATCHDIR/project/mintpy/geodmod(gbis)_startDate_endDate/. And you can set your favorite outdir
  4. About generate_horzvert.py, I am thinking that it may be unnecessary to use save_for_modelling.py to process it. Because the input of this script must be two tracks. I will modify the environment setting in this script tomorrow.
ranneylxr commented 5 years ago

The structure of directory in $MODELDIR (outdir you set) is changed:

  1. for geodmod and gbis, the output will be: DarbandikhanSenAT73/geodmod_startdate_enddate/ DarbandikhanSenDT80/geodmod_startdate_enddate/
  2. for generate_horzvert, the output will be: DarbandikhanSen/horzvert/startdate_enddate/