search

insarlab / MintPy

Miami InSAR time-series software in Python
https://mintpy.readthedocs.io
Other
605 stars 259 forks source link

Adding topsStack, ssara support to tropo_pyaps3.py ? #253

Closed falkamelung closed 4 years ago

falkamelung commented 4 years ago

Hi @yunjunz @hfattahi @bhuvankumaru @mirzaees , I generally skip atmospheric corrections because the download of the atmospheric models can take very long. We should start tropo_pyaps3.py at the beginning of the SSARA download or topsStack processing. We (Xiaoran @ranneylxr ) are considering to add options to read dates and times from the SAFE_file_list.txt or the ssara_federated_query.py --print. Given that we have --date-list we would do:

tropo_pyaps3.py  --SAFE-list SAFE_file_list.txt
tropo_pyaps3.py  --ssara_list ssara_listing.txt

Any thoughts?

Heres the frames that would be supported:

cat SAFE_files.txt 
/projects/scratch/insarlab/famelung/BolnaySenDT150/SLC/S1B_IW_SLC__1SDV_20170619T234755_20170619T234823_006126_00AC37_040E.zip
/projects/scratch/insarlab/famelung/BolnaySenDT150/SLC/S1A_IW_SLC__1SDV_20160712T234822_20160712T234852_012122_012C64_F3BA.zip
/projects/scratch/insarlab/famelung/BolnaySenDT150/SLC/S1A_IW_SLC__1SDV_20161203T234810_20161203T234836_014222_016FD8_3BAB.zip
/projects/scratch/insarlab/famelung/BolnaySenDT150/SLC/S1A_IW_SLC__1SDV_20170120T234807_20170120T234834_014922_0185A3_2BCC.zip

cat ./ssara_listing.txt
//login4/projects/scratch/insarlab/famelung/WenchuanSenDT62/SLC[1086] cat ./ssara_listing.txt
Running SSARA API Query:  https://web-services.unavco.org/brokered/ssara/api/sar/search?platform=SENTINEL-1A%2CSENTINEL-1B&relativeOrbit=62&start=%3D2014-01-01&processingLevel=L0%2CL1.0%2CSLC&intersectsWith=Polygon%28%28102.10+30.40%2C+102.10+31.60%2C+105.40+31.60%2C+105.40+30.40%2C+102.10+30.40%29%29
SSARA API query: 34.263800 seconds
Found 292 scenes
wget -O dem.tif "http://ot-data1.sdsc.edu:9090/otr/getdem?north=33.824873&south=28.222164&east=105.754675&west=101.805635&demtype=SRTMGL1_E"
gdal_translate -of GMT -ot Int16 -projwin 101.805635 33.824873 105.754675 28.222164 /vsicurl/https://cloud.sdsc.edu/v1/AUTH_opentopography/Raster/SRTM_GL1_Ellip/SRTM_GL1_Ellip_srtm.vrt dem.grd
ASF,Sentinel-1A,2759,2014-10-09T23:03:54.000000,2014-10-09T23:04:21.000000,62,2952,2952,IW,NA,DESCENDING,R,VV,https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SSV_20141009T230354_20141009T230421_002759_003198_8E1D.zip
ASF,Sentinel-1A,2759,2014-10-09T23:04:19.000000,2014-10-09T23:04:47.000000,62,2981,2981,IW,NA,DESCENDING,R,VV,https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SSV_20141009T230419_20141009T230447_002759_003198_7263.zip
ASF,Sentinel-1A,3109,2014-11-02T23:03:54.000000,2014-11-02T23:04:21.000000,62,2952,2952,IW,NA,DESCENDING,R,VV,https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SSV_20141102T230354_20141102T230421_003109_00390D_8B4F.zip

Description of the desired feature

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you have considered

Additional context

Are you willing to help implement and maintain this feature? Yes/No

yunjunz commented 4 years ago

Hi @falkamelung, downloading can be initiated early in your processing by calling the dload_grib_files from tropo_pyaps3.py. As you can see it from the code, the input will be the date, time of your datasets and optionally, the SNWE of your area of interest.

Adding some functions inside tropo_pyaps3.py to prepare the arguments above from the input text file sounds good to me. Of course, you could also write a simple wrapper script in your processing recipe to do it.

falkamelung commented 4 years ago

@yunjunz Do you mean, you prefer we DO NOT modify tropo_pyaps3.py but rather create a new script which is not part of MintPy? I thought this would be a useful addition. But you need to decide whether MintPy scripts should support non-MintPy formats. Obviously we would add a --download_only option:

tropo_pyaps3.py --download_only --SAFE-list SAFE_file_list.txt
yunjunz commented 4 years ago

Hi @falkamelung, adding it to tropo_pyaps3.py is preferred with the option name you suggested.

The current implementation supports download only, when geometry file is not input.

falkamelung commented 4 years ago

Thank you! And to skip tropospheric correction if the atmospheric model download is not complete, we probably would add an option 

mintpy.troposphericDelay.method = pyaps-if-models-downloaded

I alway got frustrated when the processing waits for atmospheric model download.

yunjunz commented 4 years ago

The ERA5 downloading seems quite good. I don’t recall to have Ling waiting. I recommend you setup a common directory for $WEATHER_DIR for the whole group on HPC. The script will check the downloaded local files and try to download when there are files missing, and try again for 2 more times if the download fails. Then it will use whatever it’s available locally to calculate the delay. This means that it will correct for tropo delay for the acquisitions with weather data and skip those acquisitions without, which are usually the latest acquisitions.

yunjunz commented 4 years ago

Hi @falkamelung, as I commented above, I don't find ERA5 downloading takes a long time. Thus, I don't find this feature urgent.

Is anyone working on the proposed functionality? If not, I will close this issue.

falkamelung commented 4 years ago

yes, I hope @ranneylxr will commit soon. With dask working the download of the models is again a big timesink

yunjunz commented 4 years ago

Just want to remind you that, for GAM data downloading purpose, it's okay to download more data than needed, which means that we don't have to worry about which acquisitions are dropped during the bounding box checking in stackSentinel.py. Therefore, one should just need to find:

  1. the list of unique acquisitions in the SAFE file
  2. the rough SNWE of the AOI

The SNWE is grabbed from the input mintpy files (time-series or geometry, whichever is available) at L242 and passed to pyaps3, so that we only download a fraction (~20 by 20 deg) of the grib file, instead of the whole world, this significantly reduced the job waiting time in ECMWF, the downloading time and the file size, as you can see below.

This SNWE functionality is used when there is a mintpy file input only. But adding the support when there is no mintpy h5 file input should be straight forward.

-rw-r--r--    1 yunjunz  staff   1.4M Mar 14 19:22 ERA5_S30_S10_W80_W60_20200216_23.grb
-rw-r--r--    1 yunjunz  staff   1.4M Mar 14 19:22 ERA5_S30_S10_W80_W60_20200228_23.grb
-rw-r--r--    1 yunjunz  staff   741K May  4 16:23 ERA5_S40_S30_E10_E30_20190303_18.grb
-rw-r--r--    1 yunjunz  staff   741K May  4 16:23 ERA5_S40_S30_E10_E30_20190315_18.grb