search

Open-EO / FuseTS

Time series Fusion toolbox integrated with openEO
https://open-eo.github.io/FuseTS/
Other
22 stars 4 forks source link

Allow the setup custom cloud mask in CropSAR_px process, as SCL might be insufficient #80

Open GriffinBabe opened 1 year ago

GriffinBabe commented 1 year ago

I am currently trying to generate NDVI signals through the CropSAR_px process in OpenEO.

My area of interest is the Kibale National Park, Uganda. It is a highly clouded area, and the SCL mask often misses entire patches of cloud, as you can see in the following examples that I found while investigating the clouds in the zone.

incomplete_scl_example_2 incomplete_scl_example_3 incomplete_scl_example_1

The average NDVI in the bounds (30.36237326201062, 0.2245859481573243, 30.382159199064937, 0.3745267195130956) for the temporal extent ['2021-01-01', '2022-12-31'] gave me the following result:

cropsar_px_ndvi

This is the NDVI of two years in a region that has two growing seasons per year. The four cycles can bee perceived, although there are sudden NDVI drops present that impact the quality of the curve, I highly suspect that this is due to the undetected clouds in the SCL mask.

Would it be possible to add an optional custom cloud mask in the CropSAR_px process?

jdries commented 1 year ago

@kvantricht from an openEO perspective this is doable, but should cropsar team perhaps investigate this particular case of (failing) cloud masking? @GriffinBabe did you look at CLP and CLD layers provided by Sentinelhub based on S2Cloudless? These should be available for that region, and would avoid requiring to run an additional UNet to generate the mask ourselves.

kvantricht commented 1 year ago

I don't think the cloud masking itself should be responsibility of cropsar_px. Having said that, I'm thinking you're showing pure SCL mask, and not the dilated version we generate within CropSAR? It would be interesting to compare that one as well. As for the custom mask, how would see such feature @jdries ? An additional UDF?

GriffinBabe commented 1 year ago

@kvantricht It's true that I haven't considered the SCL masking in the examples here. But in some examples (the 3rd is especially strong) large patches of cloud masks are entirely missing, not sure that the mask dilation is enough here, the area is very large, here the image was coarsened by a factor x10.

What I'm doing at the moment is to do the union of the SCL mask and the custom mask. But maybe I should also apply the mask dilation on this, you are right

GriffinBabe commented 1 year ago

@jdries Are those layers available using the SentinelHub provider but from the Terrascope OpenEO implementation?

jdries commented 1 year ago

Yes, those bands can be found in SENTINEL2_L2A_SENTINELHUB: image

GriffinBabe commented 1 year ago

I'm coming back on this issue as I recently wrote the report about the first use case of the experimentation plan.

This following figure compares the NDVI of a point obtained through the cropsar_px in two different ways:

Picture1

Points are at locations: [(30.38406 0.49868), (33.05291 0.46595), (31.57619 1.77082)] for the temporal extent: ['2021-01-01', '2022-12-31']

As you can see there are still some unexpected NDVI drops for discontinuous dates, but much less. Adding the possibility of specifying a custom SCL mask would allow the user to improve the quality of the outputted curves.

@JanssenBrm

kvantricht commented 1 year ago

Hi Darius,

I'd love to get an update from you regarding your experience with pixel-based cropsar. Both the good and bad stuff. I have to write a blog on it, so better know from one of the early adopters like you what to expect. Can we discuss next week?

Best, Kristof

From: Darius Couchard @.> Sent: Wednesday, 16 August 2023 12:00 To: Open-EO/FuseTS @.> Cc: Kristof Van Tricht @.>; Mention @.> Subject: Re: [Open-EO/FuseTS] Allow the setup custom cloud mask in CropSAR_px process, as SCL might be insufficient (Issue #80)

I'm coming back on this issue as I recently wrote the report about the first use case of the experimentation plan.

This following figure compares the NDVI of a point obtained through the cropsar_px in two different ways:

[Picture1]https://user-images.githubusercontent.com/23267782/260969001-8fc5546e-667f-4daa-a099-a089df7da0c2.png

Points are at locations: [(30.38406 0.49868), (33.05291 0.46595), (31.57619 1.77082)] for the temporal extent: ['2021-01-01', '2022-12-31']

As you can see there are still some unexpected NDVI drops for discontinuous dates, but much less. Adding the possibility of specifying a custom SCL mask would allow the user to improve the quality of the outputted curves.

@JanssenBrmhttps://github.com/JanssenBrm

- Reply to this email directly, view it on GitHubhttps://github.com/Open-EO/FuseTS/issues/80#issuecomment-1680316266, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AP2XQ3CFUPS55BUI2NYSXSDXVSK3JANCNFSM6AAAAAAXZS7ZUA. You are receiving this because you were mentioned.Message ID: @.**@.>>

VITO Disclaimer: http://www.vito.be/e-maildisclaimer

JanssenBrm commented 1 year ago

As discussed during the review of this issue, adding a custom cloud mask would have a big impact both on the usability and the implementation of the service. Supporting a custom cloud mask as an input requires the provided cloud mask to be compatible with the CropSAR implementation. In any other case, CropSAR could interpret the mask wrong, calculating incorrect results.

As an alternative, more user-friendly solution, CropSAR could support a cloud mask selection as an input parameter. The user can use this selection to choose the cloud mask that should be applied. For example, CropSAR could add support for S2Cloudless and provide the user with the possibility to choose between the default cloud mask or the new option.

Patrick1G commented 1 year ago

Hello @JanssenBrm @jdries et al.

I asked Daniel from Sinergise if the S2Cloudless could be added as a collection to openEO. He responded that this is provided as a precomputed layer in the SH S2L2A collection. See details below. Could this help for integrating it into the fuseTS workflows?

Not sure if this is exactly what you are looking for, but we do already have a S2 collection (SENTINEL2_L2A_SENTINELHUB) available which includes 2 bands (CLP = cloud probabilities, CLM = cloud mask) that were computed using s2cloudless at a fixed 160m resolution (more info here.

Can you please check if this is helpful for the projects or otherwise provide me some more details on what exactly you would like to be available?

jdries commented 1 year ago

@GriffinBabe did we use that when testing the various cloud masking options? (It's been available for quite some time and indeed works.)