search

EO-College / cubes-and-clouds

This is the official repository for the online course 'Cubes & Clouds'
Creative Commons Attribution 4.0 International
32 stars 12 forks source link

Sentinel-2 masking: performance optimization #26

Closed jdries closed 11 months ago

jdries commented 11 months ago

In this notebook: https://github.com/EO-College/cubes-and-clouds/blob/main/lectures/3.1_data_processing/exercises/31_data_processing.ipynb

The cloud masking of Sentinel-2 results in this 'diamond shaped' process graph: image

This works, but is not the most efficient approach, because to create the mask, the backend also needs to load the other bands already. Cloud masking often has the effect that full chunks can be discarded, so my general recommendation is to create a separate cube for the SCL band, convert it to a mask, and then load the other bands and apply the mask. The backend can then only load the chunks of data that are not fully masked.

przell commented 11 months ago

Hi Jeroen, thanks for the suggestion. We'll adapt accordingly! ping @Ruphai

clausmichele commented 11 months ago

@jdries your suggestion has been integrated!