S2 cfmask product snafu

[lledoux]

I'm not even sure if this is something we can manipulate, or if it's worth it (as it seems to happen only to a few tiles). I used my virtual environment /scratch/lledoux/cmsindo/test-venv/ on rio to process Sentinel 2 data, and then create a project directory. I have noticed on a couple images that the cfmask product seems to be off. The product is not off for every date in the directory, just a couple. An example of it is seen below:

This is a reflectance image, the clouds are the white (basically the entire image):

And this is the cfmask product (where the black pixels have a value of "1" (valid data) (separate question- why is the northern part all gray?):

This particular product is /projects/NASsarag1/data/thaibin/2017_sentinel2/thaibinh_32648_30.0x30.0_sentinel2/0/2017094_S2A_cfmask.tif

And this is the same ref product, with gips_mask applied:

[lledoux]

Update- that gray in the northernmost tip of the area is due to the overlap with another sentinel2 tile, and the cfmask product from that different tile says that there is no valid data in the whole tile (which is true):

The ref products of the two tiles (clouds are white, nearly no valid pixels):

The cfmask products of the two tiles (valid "1s" are black):

[bhbraswell]

For S2 do we make the mask or are we using a mask that comes with the data?

[lledoux]

That's what I can't remember...

[ircwaves]

$0.02 -- cloud detection is usually imperfect. Both of those tiles look like >90% cloud, don't they? gdalinfo can get you the info*. We haven't worked out how to wire in the cloud filter to the command-line interface yet, but we could pull all imagery that is this cloudy?

With respect to the question of how it is generated: we use the CFmask implementation from the folks down in Christchurch (see setup.py for the exact version and link).

* Here's how to view the cloud cover assessment of a sentinel 2 asset:

icooke@north:~$ gdalinfo /vsizip/S2A_MSIL1C_20170413T153911_N0204_R011_T19TCH_20170413T154415.zip 2>/dev/null \
    | grep MTD_MSIL1C.xml
       /vsizip/S2A_MSIL1C_20170413T153911_N0204_R011_T19TCH_20170413T154415.zip/S2A_MSIL1C_20170413T153911_N0204_R011_T19TCH_20170413T154415.SAFE/MTD_MSIL1C.xml
icooke@north:~$ gdalinfo /vsizip/S2A_MSIL1C_20170413T153911_N0204_R011_T19TCH_20170413T154415.zip/S2A_MSIL1C_20170413T153911_N0204_R011_T19TCH_20170413T154415.SAFE/MTD_MSIL1C.xml \
    | grep CLOUD
CLOUD_COVERAGE_ASSESSMENT=100.0

[lledoux]

I tried the above recommendation, and received this error:

lmelendy@rio:/titan/data/sentinel2/tiles/48QXJ/2017094$ gdalinfo /vsizip/S2A_MSIL1C_20170404T034151_N0204_R018_T48QXJ_20170404T034147.zip/S2A_MSIL1C_20170404T034151_N0204_R018_T48QXJ_20170404T034147.SAFE/MTD_MSIL1C.xml
ERROR 1: Cannot find Query_Options.Band_List
gdalinfo failed - unable to open '/vsizip/S2A_MSIL1C_20170404T034151_N0204_R018_T48QXJ_20170404T034147.zip/S2A_MSIL1C_20170404T034151_N0204_R018_T48QXJ_20170404T034147.SAFE/MTD_MSIL1C.xml'.

But it works with

gdalinfo /vsizip/S2A_MSIL1C_20170404T034151_N0204_R018_T48QXJ_20170404T034147.zip/S2A_MSIL1C_20170404T034151_N0204_R018_T48QXJ_20170404T034147.SAFE/GRANULE/L1C_T48QXJ_A009310_20170404T034147/MTD_TL.xml | grep CLOUD

[ircwaves]

Another amazing inconsistency in the sentinel2 data formatting. Specific metadata stored in one XML file on one day, and in another XML file on another day.