ATL08 PhoREAL Flags Interpolation Issue

[ejguenther]

In the ATL08 PhoREAL Sliderule functionality, it appears that certain flag interpolations are incorrect. When the function icesat2.atl08p(parms, keep_id=True) is called in the Sliderule-Python Client, when using atl08_fields that use ints, it appears that the interpolation results in erroneous values. For example, calling the msw_flag or segment_landcover results in this.

Here is an example for segment_snowcover, the flags should only be 0,1,2,3,4 but we have a range outside of that:

Other examples here impact the MSW Flag:

This also appears with the Landcover Flag. Other int fields that are not related to field interpolation seem to be OK, here is an example of the Spot Number, which appear nominal:

Another note: segment_landcover results in the incorrect values, though the landcover field that is default in the ATL08 output appears normal. However, segment_snowcover also results in incorrect values as well as snowcover.

[jpswinski]

@ejguenther Can you send me the full set of parameters you are using for your request, along with the polygon you are processing (or the granule name, if that is what you are using)?

I am not able to recreate this issue with the tests that I have, so I want to use what you are using to recreate the problem.

[ejguenther]

Here are the parameters I am using:

parms = {
    "poly": poly_out,
    "t0": '2021-01-01T00:00:00Z',
    "t1": '2021-03-01T00:00:00Z',
    "srt": icesat2.SRT_LAND,
    "len": 30,
    "res": 30,
    "pass_invalid": True, 
    "atl08_class": ["atl08_ground", "atl08_canopy", "atl08_top_of_canopy"],
    "atl08_fields": ["canopy/h_canopy_uncertainty","h_dif_ref","msw_flag","sigma_topo","segment_landcover","canopy/segment_cover","segment_snowcover","terrain/h_te_uncertainty"], #'segment_cover' and 'h_canopy_uncertainty' need to be added
    #"atl08_fields": ["h_dif_ref","msw_flag","sigma_topo","segment_landcover"],
    "phoreal": {"binsize": 1.0, "geoloc": "center", "above_classifier": False, "use_abs_h": False, "send_waveform": True}
}

Where poly_out is this KML which can be downloaded here: https://utexas.box.com/s/k7nxfgbt0envla82elcg0dv2bw2y7pqs

[jpswinski]

@ejguenther I think I found the bug (at least I definitely found a bug, hopefully it is this bug) - see the commit above 14a8ffb for the exact change.

When I calculated the index into the land_segments group variables for each photon (which I need in order to grab the correct ancillary data value), I wasn't accounting for the case where there was a gap at the end of the set of segments. (i.e. the fact that not all ATL03 segments are present in the ATL08 data; sometimes a segment is not used).

When that happened, my ATL08 segment index was set to one element past the end of the array. The code that later pulls the ancillary data out and associates it with each photon does an array bounds check, and because the index is out of bounds, it never reads or writes any value - leaving the value in the ancillary fields array to be left uninitialized.

We are then seeing those uninitialized values (somewhat random in that it is whatever was left over before in that memory space) in our data.

[ejguenther]

Tested this on a few queries on my end and there does not appear to be any more out of range values, in addition the fields such as segment_landcover and landcover and segment_snowcover and snowcover are now in agreement. I'd consider this issue resolved.