ICESAT-2HackWeek / h5cloud

BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

Create controlled ATL03 HDF5 test file #27

Open asteiker opened 8 months ago

asteiker commented 8 months ago

To better control these performance tests, we ought to create stripped down ATL03 files based on information from #23 and #26. The goal is to better control for the complexities in the native data. We can then reduce the noise as we investigate switches in the HDF5 library to increase performance in the cloud.

### Tasks
- [x] Determine key variables, including quality flags, needed for higher level products.
asteiker commented 8 months ago

@andypbarrett to provide key variables, including quality flags, needed for higher level products. Utilize ATBD and earthdata-varinfo to inform what information is needed for a controlled ATL03 HDF5 test file.

andypbarrett commented 7 months ago

I've looked at ATL07 and ATL06. Below are a list of candidate variables. I've left out geophysical corrections that are already included in _hph. Paths are from ATL03 data dictionary. Variable lists are from table 2 in ATL07 ATBD and table 5-1 in ATL06 ATBD.

11 out of the 13 variables in /gtx/heights are used by one or both of ATL07 and ATL06, and 12 out of 37 variables from /gtx/geolocation are used.

ATL03 Variable Description Data type Data Rate Source
_/gtx/heights/deltatime Elapsed GPS seconds Double(:) Photon Detection Rate 6 + 7
_/gtx/heights/dist_phacross Distance off RGT Float(:) Photon Detection Rate 6 + 7
_/gtx/heights/dist_phalong Distance from equator crossing Float(:) Photon Detection Rate 6 + 7
_/gtx/heights/hph Photon WGS84 height Float(:) Photon Detection Rate 6 + 7
_/gtx/heights/latph Latitude Double(:) Photon Detection Rate 6 + 7
_/gtx/heights/lonph Longitude Double(:) Photon Detection Rate 6 + 7
_/gtx/heights/pce_mframecnt PCE major frame counter UINT(:) Photon Detection Rate 6
_/gtx/heights/ph_idchannel Receive channel id UINT(:) Photon Detection Rate 6
_/gtx/heights/ph_idpulse laser pulse counter id UINT(:) Photon Detection Rate 6
_/gtx/heights/qualityph Photon quality INT Photon Detection Rate 7
_/gtx/heights/signal_confph Photon signal confidence INT Photon Detection Rate 6
_/gtx/geolocation/deltatime Elapsed GPS seconds DOUBLE(:) Segment rate 6
_/gtx/geolocation/ph_indexbeg Index of first segment photon INT Photon Detection Rate 6
_/gtx/geolocation/podppdflag Geolocation quality INT Photon Detection Rate 6
_/gtx/geolocation/segment_distx Segment distance from equator Double(:) Segment rate 6
_/gtx/geolocation/segmentid along-track segment id INT(:) Segment rate 6 + 7
_/gtx/geolocation/segmentlength along-track segment length Double(:) Segment rate 6 + 7
_/gtx/geolocation/segment_phcnt Number of photons INT(:) Segment rate 6
_/gtx/geolocation/sigmaacross across-track geolocation accuracy Float(:) Segment rate 6 + 7
_/gtx/geolocation/sigmaalong along-track geolocation accuracy Float(:) Segment rate 6 + 7
_/gtx/geolocation/sigmah height uncertainty (Float(:) Segment rate 6
_/gtx/geolocation/surftype Surface type INT(:, 5) Segment rate 7
_/gtx/geolocation/velocitysc spacecraft xyz velocity Float(:,3) Segment rate 6
asteiker commented 7 months ago

We determined that we will grab all variables within the /heights groups for each beam group. A new task was added to actually perform the subsetting using icepyx.

asteiker commented 1 month ago

This may still be helpful to determine core variables needed to create a derived product if we were to reformat to something like geoparquet.