Ingest of High-res data

[falkamelung]

The high resolution data ((i) Persistent scatterer (PS) -- small file because most data are masked out and (ii) Distributed Scatterer (DS) - large file a small file) are at

cd /data/HDF5EOS/Miami2SenAT48/minopy/single_reference_network

Persistent Scatterer (PS). To create 'S1_IW3_048_0081_0082_20200110_20220429_PS.he5`

save_hdfeos5.py timeseries_demErr.h5 --tc temporalCoherence.h5 --asc avgSpatialCoh.h5 -m  ../maskPS.h5 -g inputs/geometryRadar.h5 -t smallbaselineApp.cfg --suffix PS
save_kmz.py velocity.h5 --mask ../maskPS.h5 -g inputs/geometryRadar.h5 --step 1 --output velocityPS.kmz

Get kmz (6MB) with https://js-104-223.jetstream-cloud.org/data/HDF5EOS/Miami2SenAT48/minopy/single_reference_network/velocityPS.kmz:

Zoom on the Surf Club hotel:

Distributed Scatterer (DS): To create 'S1_IW3_048_0081_0082_20200110_20220429.he5`

save_hdfeos5.py timeseries_demErr.h5 --tc temporalCoherence.h5 --asc avgSpatialCoh.h5  -m  maskTempCoh.h5 -g inputs/geometryRadar.h5 -t smallbaselineApp.cfg 
save_kmz.py velocity.h5 -m  maskTempCoh.h5 -g inputs/geometryRadar.h5 --step 1 --output velocity.kmz

Get kmz (120 MB)with https://js-104-223.jetstream-cloud.org/data/HDF5EOS/Miami2SenAT48/minopy/single_reference_network/velocity.kmz

Zoom on the Surf Club hotel:

[stackTom]

Will try with the above.

Question - how big do these points have to be? Since these new datasets don't have x step and y step.

[falkamelung]

It is actually not so simple. In range (east-west) direction the pixel spacing varies significantly for high rise buildings, so there is no precise x_step. So just do 5 m diameter circles (when highly zoomed in). A scale showing the approximate diameter would be helpful but is not critical (see below).

As you saw from Yunjun's email there will be a new h5 dataset name maskPS (once I have added it). So I would suggest an option to use a different h5 datatset for mask. How about 'hdfeos5_2json_mbtiles.py --mask maskPS.h5' (alternatively -m maskPS.h5 to use a specific mask). (default would be current mask)

On symbol size: In addition it will be good to have the option to change the size manually but this is also not critical. I can see two options for this: (i) to select a size in meters (somewhere where the actual 'actual size' button is, only if we know how to properly display a size given in meters). (ii) To have a button to repeatedly half/double the symbol size in the Layer window (we may want to move it to the display window at a later stage if used a lot (I originally wanted to use it a lot but after having the other things figured out I am not sure and first need to try).

On rectangles: For geocoded data (when x_step, y_step is there) It would be good to have an 'hdfeos5_2json_mbtiles.py --rectangles [-r]` option. If there is no performance loss we may want to make this the default - I have to try first. It is not critical though. But I believe you have figured this out already.

On scale showing the symbol size in meters If we have proper control on the symbol size it would be good to somehow display it (e.g. appearing only for zoom level>15). It is accurate only for geocoded data. For radar geometry data only the azimuth (north-south) direction is accurate, but a scale would still be useful although approximate.

[stackTom]

It is actually not so simple. In range (east-west) direction the pixel spacing varies significantly for high rise buildings, so there is no precise x_step. So just do 5 m diameter circles (when highly zoomed in). A scale showing the approximate diameter would be helpful but is not critical (see below).

As you saw from Yunjun's email there will be a new h5 dataset name maskPS (once I have added it). So I would suggest an option to use a different h5 datatset for mask. How about 'hdfeos5_2json_mbtiles.py --mask maskPS.h5' (alternatively -m maskPS.h5 to use a specific mask). (default would be current mask)

On symbol size: In addition it will be good to have the option to change the size manually but this is also not critical. I can see two options for this: (i) to select a size in meters (somewhere where the actual 'actual size' button is, only if we know how to properly display a size given in meters). (ii) To have a button to repeatedly half/double the symbol size in the Layer window (we may want to move it to the display window at a later stage if used a lot (I originally wanted to use it a lot but after having the other things figured out I am not sure and first need to try).

On rectangles: For geocoded data (when x_step, y_step is there) It would be good to have an 'hdfeos5_2json_mbtiles.py --rectangles [-r]` option. If there is no performance loss we may want to make this the default - I have to try first. It is not critical though. But I believe you have figured this out already.

On scale showing the symbol size in meters If we have proper control on the symbol size it would be good to somehow display it (e.g. appearing only for zoom level>15). It is accurate only for geocoded data. For radar geometry data only the azimuth (north-south) direction is accurate, but a scale would still be useful although approximate.

We can't do rectangles in hdfeos5_2json_mbtiles. It needs to be done in the browser, as ingesting rectangles won't work (see earlier example in another issue which shows that we see white space between points).

[stackTom]

See here. It seems to work: http://129.213.120.104/start/25.8768/-80.1212/18.3722?flyToDatasetCenter=false&startDataset=S1_IW3_048_0081_0082_20200110_20220429_PS&pointLat=25.87645&pointLon=-80.12170&minScale=0.4&maxScale=-0.4&startDate=20200110&endDate=20220429

However, there are very small discrepancies between the location of the points and the location seen in the kmz. I have narrowed this down after a lot of debugging to either 1) tippecanoe or 2) mbtiles format. I will need to ask mapbox and/or the tippecanoe creators why the coordinates are not ported over identically inside the mbtiles (they seem to be slightly off when looking at the raw numbers).

[falkamelung]

It looks to me that insarmaps displays less points then the kmz file??? The chunk file does seem to have the same number of points as the *kml, though.

http://129.213.120.104/start/25.8771/-80.1214/19.3124?flyToDatasetCenter=false&startDataset=S1_IW3_048_0081_0082_20200110_20220429_PS&pointLat=25.87648&pointLon=-80.12158&minScale=0.4&maxScale=-0.4&startDate=20200110&endDate=20220429

[stackTom]

I think the different coordinates and the less points is a compromise we had to take for old style datasets. I have changed the settings for tippecanoe, and now we have the correct coordinates and number of points. I think we should have a high res mode in the ingest scripts. Do you think this should be a flag passed on the command line, or an attribute in the S1 file?

Anyways, here is the new version with all points and correct coordinates: http://129.213.120.104/start/25.8935/-80.0931/11.9734?flyToDatasetCenter=false&startDataset=S1_IW3_048_0081_0082_20200110_20220429_PS&minScale=0.4&maxScale=-0.4

[stackTom]

PS make sure to clear your cache before viewing it.

[falkamelung]

Oh, that looks great!! It should be able to detect whether we want high-res depending on whether data are in geo or radar coordinates (x_step given), isn't it? But it would be good to have an --high-res and --normal (or --low-res) option to override this.

[falkamelung]

That is great. OF course

[stackTom]

Hmmm. It appears that my new tippecanoe options are in fact working. It seems that I was just using an old s1 file due to the fact that the new one wasn't uploading correctly due to the bug I just fixed over chrome remote desktop (php upload size too small after darren updated php).

Check distributed scatter here. It looks good to me: http://129.213.120.104/start/25.8774/-80.1206/16.8750?flyToDatasetCenter=false&startDataset=S1_IW3_048_0081_0082_20200110_20220429&minScale=1&maxScale=-1

Can you look and let me know what you think.

[falkamelung]

This looks great! Thank you!

Yes, proper messaging about ingest success and failure is important.

Now I remember what we wanted to do. Output something about memory/diskspace required/avialable for each ingest.

[falkamelung]

Closing as it seems to work now.