sentinel3: cloud masking does not seem to work

[jdries]

con = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc()

mask = con.load_collection(
    "SENTINEL3_SLSTR_L2_LST",
    spatial_extent={"west": -100, "south": 40, "east": -90, "north": 60},
    temporal_extent=["2020-08-03", "2020-08-10"],
    bands=["flags_in:confidence_in"]
)

mask = mask >= 16384

datacube = con.load_collection(
    "SENTINEL3_SLSTR_L2_LST",
    spatial_extent={"west": -100, "south": 40, "east": -90, "north": 60},
    temporal_extent=["2020-08-03", "2020-08-10"]
)
datacube.mask(mask)
datacube.aggregate_temporal_period(period="day",reducer="max").execute_batch("/tmp/forum676-us_masked.nc")

[clausmichele]

@jdries we are also interested to use Sentinel-3 with CDSE and also interested to see if there's something new for cloud masking. Is this issue related to the fact that the mask is not satisfying or that it doesn't mask at all despite there's an area to be masked?

[jdries]

Not entirely sure yet what's wrong. I had the impression that I was still seeing clouds after applying the mask, but have to figure out the reason. I also still have to learn how good or bad the included cloud mask is.

[clausmichele]

I guess that the cloud mask is just bad, it's still a scientifically relevant issue cloud masking for Sentinel-3!

[jdries]

this one is a lot better now. I also heard that cloud masks are better from octobre 2020 onwards.

Here's a somewhat working example:

con = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc()

    mask = con.load_collection(
        "SENTINEL3_SLSTR_L2_LST",
        spatial_extent={"west": -100, "south": 40, "east": -90, "north": 60},
        temporal_extent=["2021-08-03", "2021-08-10"],
        bands=["confidence_in"]
    )

    mask = mask >= 16384

    datacube = con.load_collection(
        "SENTINEL3_SLSTR_L2_LST",
        spatial_extent={"west": -100, "south": 40, "east": -90, "north": 60},
        temporal_extent=["2021-08-03", "2021-08-10"],
        bands=["LST"]
    )

    lst_celsius = datacube.band("LST_in:LST") - 273.15

    from openeo.processes import if_
    masked = lst_celsius.add_dimension("band", "lst", type="bands").merge_cubes(
        mask.add_dimension("band", "mask", type="bands")).apply_dimension(dimension="band",
                                                                          process=lambda x: if_(x.array_element(1) == 0,
                                                                                                x.array_element(0)))

    aggregated = masked.aggregate_temporal_period(period="day", reducer="max")
    aggregated.download("/tmp/sentinel3_composite.nc")

[clausmichele]

@jdries is it normal that I can't connect to the openEO back-end with the link you provided?

[jdries]

I changed the endpoint, please try again

[clausmichele]

I tried with two different AOIs over the Alps but the batch job stops with an error. However, in the logs no error is reported. Big AOI Alps, job id: j-240229e5668a4d2ca3d0bdf1b250e4fa Smaller AOI, job id: j-240229b373964f29bfedb32b3a98066b

Edit: sync call works instead