ceos-org / ceos-ard

Repository for CEOS Analysis Ready Data (CEOS-ARD), including Product Family Specifications (PFSs)
9 stars 0 forks source link

'No Data' Definition #4

Open mattsymbios opened 3 months ago

mattsymbios commented 3 months ago

No Data (2.2) is differently defined in the AR PFS. This should be aligned with the other optical PFS to ensure consistency across ARD data.

Matthias' proposal: Replace “(‘empty pixels’)” with “(e.g., ‘empty pixels/invalid observation/below noise floor’)” in all PFS (except AR)

The empty pixels terminology is problematic.

We will raise at LSI-VC-15 in the combined optical PFS discussion - and also note the need to harmonise this terminology with the SAR side too.

strobpr commented 3 months ago

I see four cases covered under this topic, which I would recommend to differentiate at threshold level:

'NoData': there is no sensor reading available for this location, e.g. outside FOV, obstruction(?), (ideally these would be distinguished -> goal)

'Invalid': an observation was acquired by the sensor, but no valid result is obtained, e.g. below noise floor, above saturation, encoding defect (ideally these would be distinguished -> goal)

'Falsified': a valid observation was acquired but it is not representing the desired measurand, e.g. cloud/snow/flooding/shadow (ideally these would be distinguished -> goal)

'Valid': these are verifyable observations of the measurand (which actually would imply that uncertainty is provided as well, which so far is only goal), e.g. ToR, LST, γ0

The first two are sensor dependent, the last two measurand dependent. The latter thus may change over the value chain,In other words a sample 'valid' e.g. in ToR, may turn 'falsified' in SR. Ideally we should provide a bit scheme for a byte binary mask that allows encoding all these options using the least possible space. The smaller they are the faster these very fundamental indicators can be transmitted. Bit decoding back to simple binary masks is fast and easy.

strobpr commented 3 months ago

Note that this issue is not just 'optical'. Should be linked with issues #7 and #17 for a holistic discussion of flags regarding data quality. Many can be generalised and streamlined across all types of observations.

sjskhalsa commented 3 months ago

The observation data could have only valid measurements and then a single FillValue, and the reason for fill would be in a separate data array, perhaps as a bit field where various conditions, including suspected cloud, could be conveyed. recall, cloud detection is never 100% accurate. the presence of thin cirrus, for example, might still yield usable surface reflectance information