Closed augustinh22 closed 1 month ago
@luukvdmeer the real question is:
is there a specific, intentional reason why nan values are no longer dropped, or is it a bug?
I don't completely understand how the all-nan timesteps occured, but in any case, the trim verb can be used to get rid of all-nan slices along a given dimension. In this way you have full control over it, rather than that the data retriever drops dimension values without you being aware of it.
A bit more context:
The problem is that the granule-footprints are indexed, not the valid-data-footprints. Thus, you may have an AOI that is in the time series sometimes completely within the image footprint, sometimes partially, sometimes not:
Here, in the second example, there would be an all-empty array, which is not useful/correct. In the "old" inference engine, there was the "dropna" function applied after loading. I would prefer this for 2 reasons:
In the best case, the storage backend would not provide these images, but since we can not rely on this, it should be - in my opinion - at least a (configurable) default to drop empty slices after loading the data.
Agree with Martin argumentation, only wanted to mention that trim works - the curves then even indicate where the observation was:
Is not solving the data amount issue since applied at the end....
Yes makes sense, this should be easy to implement
@whisperingpixel @augustinh22 @dtiede The datacube classes in semantique now have the configuration parameter "trim". If True, each retrieved array will be trimmed, meaning that all dimension coordinates with only missing/invalid data are removed. For the spatial dimensions this happens only at the edges, to maintain regularity.
After your explanations I set the default for this parameter to True, think that makes most sense. Therefore, nothing needs to change in Sen2Cube (except updating semantique :)
To get the "old" behaviour:
import semantique as sq
import datacube as dc
import json
#Connecting to the cube
s2c = dc.Datacube()
dcconfig = {
"trim": False
}
with open("../layouts/factbase_austria.json", "r") as file:
cube = sq.datacube.Opendatacube(json.load(file), connection = s2c, **dcconfig)
Description
Recipe execution retrieves and then continues to drag along all timesteps for which every value has no data (i.e. nan).
In certain situations, this produces results with many nan values (e.g. a Sentinel-2 imagery granule observed by multiple acquisition orbits loaded via an opendatacube connection, whereby any observation covering any part of the granule that contains the AOI returns "nan" values if no valid values are present (i.e. outside of the acquisition swath, but still in the granule/image bounds)).
Reproducible example
The above query uses the following geojson, referred to as "area-of-interest_id68051.geojson", which happens to be an AOI in a Sentinel-2 granule (33UUP) that is observed by 3 different acquisition swaths, but the AOI is located in the middle, where only one swath captures it:
This means that the results for looking at the vegetation entity from 01.09.2021 until 30.09.2021 results in 17 timesteps even though 12 of them are completely empty and full of "nan" values BEFORE analysis ever begins:
While it is possible that "nan" values could be somehow introduced by some recipe, in this case it isn't what happend so if we apply a dropna over time for where all observations are "nan", we get the result as only 5 timesteps:
Expected behavior
Timesteps that have no valid observations (i.e. where all are "nan") get dropped after being loaded from an OpenDataCube connection and thus explicitly excluded from being dragged along the analysis chain.
(Note: this, however, may be undesired in the name of keeping functionality generic and comparable across connection types looking foward. Open for discussion.)
Environment information
Example in sen2cube.at inferences
Additional context
This used to be included in a much earlier, toddler version of semantique known as the "inferenceengine" using xarray.Dataset.dropna: https://github.com/whisperingpixel/iq-inferenceengine/blob/2846641407cf9a997b0e306e2f5e86db22bfcfb7/inferenceengine/tools/odctools.py#L56
It could be implemented using xarray.DataArray.dropna() maybe somewhere here: https://github.com/ZGIS/semantique/blob/80542d2ca95b046ced7f765446c21ce783670d23/semantique/datacube.py#L363C43-L363C43