Open scottyhq opened 5 months ago
@scottyhq - I don't know anything about this repository... I heard about this issue ticket via @alhandwerger.
I have created workflows for other OPERA packages to generate the products locally. They are not pretty or elegant. Most are WIP.
https://github.com/OPERA-Cal-Val/dswx-hls-pst-workflow/blob/dev/0-Run-DSWX-HLS.ipynb https://github.com/OPERA-Cal-Val/dswx-s1-workflow-pst/blob/dev/0_DSWx-SAR-Workflow.ipynb
Basically, I have created software to pull data from various aws public buckets and stitch the tiles together including:
https://github.com/ACCESS-Cloud-Based-InSAR/dem-stitcher https://github.com/OPERA-Cal-Val/tile-mate
I don't know about the TEC data.
I have never created the official burst database... Know this is the relevant software: https://github.com/opera-adt/burst_db
Btw - I use your gists like once a month or more. Last week - I used this to help write a proposal. We still don't have historical OPERA RTC data either :(
Thanks for the pointers @cmarshak! With @scottstanie 's updates to the opera-db repository I'm now able to run s1_cslc.py
and generate a CLSC, but I'm still unable to match the standard product through ASF. I suspect there may be a difference in either the DEM i'm using, or some software version difference (I've at least matched isce3=0.15.0 and compass=0.5.4)?
For opera_burst_database.sqlite3
I've used opera-burst-bbox-only.sqlite3
which has the following structure with UTM coordinates:
FID Column = rowid
burst_id_jpl: String (0.0)
epsg: Integer (0.0)
xmin: Real (0.0)
ymin: Real (0.0)
xmax: Real (0.0)
ymax: Real (0.0)
You can get the needed TEC file for the above example from NASA CDDIS wget --auth-no-challenge https://cddis.nasa.gov/archive/gnss/products/ionex//2023/280/JPL0OPSRAP_20232800000_01D_02H_GIM.INX.gz
Finally, for the dem, I've just used sardem --bbox -123.0 45.0 -120 48.0 --data COP
With all the auxiliary files now specified, the workflow completes and the file structure is as expected but the array values do not quite match the standard product. linear amplitude difference plotted below along with the full log
I did notice the log reports ERROR 6: IH5:::ID=360287970189640723: Dataset does not support the SetSpatialRef() method.
a number of times. I think that is coming from ISCE3 but not sure...
Hey Scott,
Two Scott's are better than one. Especially two highly capable ones!
In regards to the error, my best guess is gdal incompatibility with the minor version 3.7--> 3.8
(though this is highly speculative; I have seen this issue in both DSWx-HLS and DSWx-S1 software). I saw bizarre errors documented in Proteus here. @gshiroma figured out that it had to with how the python gdal modified some of the write API (tiff driver in particular). The quick fix in the linked case was putting a cap on the environment.
Happy adventures!
You're right that the DEM could be different:
sardem
- since I first made that awhile back when the processor I was using only accept int16
data, the the default is int16
even for --data COP
. If it's quick to check, I'd be curious if adding --output-format GTiff --output-type float32
would do. I don't think it would change the prominent mountain-ish features thoughAs far as the Dataset does not support the SetSpatialRef()
, that's an annoyance that is on the internal github's issue list to remove, but is not indicative of a real workflow error.
Just checking- you have enabled: true
for the "corrections" section of the YAML? Not sure if it would be a different of applying one of the corrections like "azimuth FM rate", which looks DEM-ish visually in the correction it applies
Thanks both for the follow up! It seems like a mismatch of DEM is the primary suspect for different outputs currently, so it would be great to get clarity on the DEM used for standard products from whoever is familiar with it.
I know that opera was using the "NISAR DEM", which was a version of the Copernicus DEM updated for use in NISAR processing. I'm not actually sure if/how different it is
Interesting, I've not yet heard of NISAR DEM... It would be great to document how it was updated, or even better just make the /home/compass_user/input_dir/dem.vrt
public? If it's pointing at public COP30 data anyways, I can't think of why it shouldn't be?
If it's quick to check, I'd be curious if adding --output-format GTiff --output-type float32 would do. I don't think it would change the prominent mountain-ish features though
Just very small differences in the output above, as expected
Just checking- you have enabled: true for the "corrections" section of the YAML?
Yes, I just copied the config file from the ['metadata']['processing_information']['runconfig']
(see first comment in issue) which has the following:
processing:
correction_luts:
azimuth_spacing: 0.028
enabled: true
Hi @scottyhq, OPERA and ASFDAAC are in conversation to make the ancillaries available where possible (depends on licensing), or provide instructions where to find them, or instructions to re-create them. We will need to walk through the process and keep you posted once it's completed.
Checked for duplicates
Yes - I've already checked
Alternatives considered
Yes - and alternatives don't suffice
Related problems
Thanks for making this code public! We're interested in creating CLSCs at different global locations back to 2014.
As a test I tried to recreate this file from ASF locally:
OPERA_L2_CSLC-S1_T013-026571-IW1_20231007T142223Z_20231009T193106Z_S1A_VV_v1.0.h5
Each HDF contains its own runconfig, but the paths to ancillary data are hardcoded and it's not clear how to get those files in place :
Describe the feature request
Add documentation on full process of creating a CSLC, in particular how to get ancillary files: