[S2GM] - Implausible values for BHR

[jscarriere]

Dear @RS-UCL,

We received this message from a user of S2GM:

"At JRC, we are working on the evaluation of the spatial variability of albedo in different urban areas. We are using 10-daily S2GM composites of BHR and DHR during 2021.

We have detected some implausible S2GM BHR values on the temporal composite from 2021-05-21 to 2021-05-31 over Marseille. Clouds cover around 80% of the city in that temporal window. However, the part of the city with valid S2 observations (1-2 during the 10-day period) shows implausible BHR values systematically above 0.9.

We hypothesize that the error could be caused by a problem in the cloud mask, or by a bug in the input datasets (Sentinel-2, VIIRS) used for that window. However, the DHR values for the same tile and period are fine.

Do you know what could be the source of this BHR problem?"

Here is the download link to the S2GM product (that you can copy in your browser):\ s2gm://czJnbS1kYXRhLmFjcmlzdC1zZXJ2aWNlcy5jb20vb3V0cHV0/73a64db9-11dc-4b77-b830-013708f0efcf/72b163aa-e4ea-480c-ae19-5657fd5feb2d/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7ImlkIjoiMGUzOTZlN2QtYTdlMy00MjI5LTgyOTUtMDYwM2Q0NDE0NTBiIn0sImV4cCI6MTcwMDcyNDI0OC44ODIsImlhdCI6MTY5ODEzMjI0OH0.KU0K1brxEP2n4nitrcIvPdhFwRrtWNO2ZIXE0Npdf-4/https:/s2gm.land.copernicus.eu

To help you find the origin of the issue, here are the two S2 scenes that were mainly used to produce the mosaic:

• S2B_MSIL1C_20210524T102559_N0500_R108_T31TFH_20230225T145115
• S2B_MSIL1C_20210524T102559_N0500_R108_T31TFJ_20230225T145115

If you are struggling to identify the issue, please feel free to ask me for support.

[ray-climate]

@jscarriere , can you please upload the figures in the temporary "Figures" directory with the following format: bhr_to_brf_band%s_type%s.png. Those files can help to better detect what causes the issue.

[jscarriere]

Hi,

Here are the logs: ucl.log Here are the figrues named "bhr_to_brf_band%s_type%s.png" :

[RS-UCL]

@jscarriere, I have pushed an update that can potentially fix the abnormal BHR value problems. Please check if that works. Thank you1

[jscarriere]

Great thanks @RS-UCL, I see that you pushed it to the main branch, could you please also push it to the s2gm_mosaic_v2 & s2gm_mosaic_dev branches ? I think that they are the ones that we use.

[RS-UCL]

@jscarriere , thanks for pointing out. Just did!

[jscarriere]

Great thanks. Just to be sure, the fix you applied is on apply_inversion.py ? We are using the multiprocessing procedure on S2GM, thus using apply_inversion_multiprocessing.py.

Do you think your commit will still fix our issue, even if it's not on apply_inversion_multiprocessing.py ?

[jscarriere]

@RS-UCL, we tried the same mosaic with your last commit, however we have the exact same result. I'm afraid that this is due to the fix made on apply_inversion.py and not on apply_inversion_multiprocessing.py.

[RS-UCL]

hi @jscarriere , sorry I should notice the updates are needed for multiprocessing. I have updated apply_inversion_multiprocessing.py. Please check if this works.

[jscarriere]

Hi @RS-UCL, we had an error on the try with the new commit, here is the error message (screenshot)

From what I see, this appears to be related to the shape of _x1filter and _y1filter that are set lines 574 & 575 in apply_inversion_multiprocessing.py (89876c9).

I notive one of the differences since the last commit (560b33d) is that you removed the two lines 571 & 572: _x1_filter = x1_filter.reshape((x1filter.size, 1)) _y1_filter = y1_filter.reshape((y1filter.size, 1))

Do you think that the issue comes from this removal ?

[jscarriere]

Hi @RS-UCL, I added the two lines myself, and it appears that we are now stuck in the While loop. I tried to understand your commit and I proposed a modification to take into account your fix by avoiding being stuck in the while loop. You can find my modification in the s2gm_mosaic_v2 branch (ddf4d5f). Do you confirm that my modification is consistent with how you wanted to fix our initial issue ?

You can find the result with the BHR (see first screenshot). It appears that the initial issue is not fixed. Do you have any idea of how we could solve it ?

You can find all the figures named "bhr_to_brf_band%s_type%s.png" here:

[jscarriere]

@RS-UCL, it appears that the overall issue comes from a low amount of cloud-free pixels. Do you know why the DHR is fine while the BHR not ? Do you think we should apply a threshold on a minimum amount of pixels, to avoid this kind of issue ?

[RS-UCL]

@jscarriere hi, sorry for the delay in response. I was working on another project last two days, and with meetings going on. I looked into the error from your screenshot, and can confirm the two lines you added are right. The BHR-BRF regression looks more reasonable than before, where the green regression plots only have one data point. Can you also upload the DHR 2D map results as well? So I can look into the difference with BHR results. Please would you also include the colourbar in the plots, so I can roughly know the value range. So far I assume the abnormal BHR retrievals are to do with limited datasets used in the regression.

[jscarriere]

Hi @RS-UCL, thanks for your answer ! I sent you an e-mail with the product through ftp.

[RS-UCL]

@jscarriere hi, I think this is indeed caused by very limited available cloud-free pixels. I checked the final cloud mask has been applied onto the data, and found after cloud filtering only very few MODIS DHRs/BHRs remain. This causes a very large variance on the regression. In addition to setting a threshold on the number of pixels for retrieval, we should also flag out abnormal BHR values (e.g. BHR>1.). How do you think?

[jscarriere]

Thanks for your answer ! I think both would be very beneficial. Do you know what would be a good threshold on the number of pixels for retrieval ? If you add the threshold, it would be great to

• Know the minimum area related to the amount of pixels needed (for exemple, if a threshold of 20 implies 0.01°x0.01°, or 0.1°x0.1° or 1°x1°). It could be in degrees or kilometres, but it will be of great help for us to fix a minimum size for the requested AOI.
• Have a new line in the logs that specifies that the procedure can't go on due to a lack of valid pixels. This will help me to then right a README in the final product that specifies to users why the procedure did not work.

And even if the threshold is not enough to remove abnormal values, it would be great to have a flag yes !

[RS-UCL]

I think it's better to use the cloud-free ratio rather than cloud-free pixels in a scence as a threshold. I updated this threshold to 20% at this moment. I expect the results of the current example will return as all nan-values for both DHR and BHR results, as the cloud-free ratio for this mosaic is below 20%. But for other scences with cloud-free ratio above this ratio, results should be unchanged. Sorry I cannot test the results on JASMIN now, but hope the updates can work. Let's me know if you think there's a better idea, and we can test it then.

[jscarriere]

Hi @RS-UCL, thanks for the update. And no worries about the tests, I can do them on my side if you can't. Regarding the threshold, can you clarify whether it is :

• the ratio between valid and cloud S2GM pixels within a VIIRS pixels?
• or the ratio between valid and cloud pixels S2GM within the whole image?
• or something else ?

If it is the first one, then it is the best solution. If not, I think it is better to apply a fixed threshold and not a ratio, because the S2GM image size may vary. The 20% valid pixels can either represents 100 or 1000 pixels depending on the size of the image. For our case, do you know how many valid pixels we have for the regression ?