search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

ERROR - First backward processing was unsuccessful, check MAJA installation #13

Closed daviddemeij closed 5 years ago

daviddemeij commented 5 years ago

I ran start_maja.py for a while and it seems like the first date seems to be processed successfully (I got the processed TIFs for the first date), but then I got an error before starting on the second image: 2019-05-09 18:46:11,809 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation.

When I check the log in the ouput directory I did find a GDAL error occuring several times at the end of the file, but I am not sure if this is the cause of the exit:

2019-05-10T09:18:51.621064 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D]  => Caching with file name vns_caching_TOAReader5.tif run in 2.8 minutes.  [vnsSentinel2L1ImageFileReaderBase.txx:GenerateTOACaching:490]
2019-05-10T09:18:51.723171 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] Caching the </home/ubuntu/efs/Start-MAJA/tmp/site_name/T33VWC/GIPP_001/in/S2B_MSIL1C_20181024T102059_N0206_R065_T33VWC_20181024T160131.SAFE/GRANULE/L1C_T33VWC_A008528_20181024T102428/IMG_DATA/T33VWC_20181024T102059_B05.jp2> image filename...  [vnsSentinel2L1ImageFileReaderBase.txx:GenerateTOACaching:477]
2019-05-10T09:18:51.723212 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] Reflectance quantification value : 0.0001  [vnsSentinel2L1ImageFileReaderBase.txx:GenerateTOACaching:478]
2019-05-10T09:18:51.766922 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] GDAL Error 1 : /home/ubuntu/efs/Start-MAJA/tmp/site/T33VWC/GIPP_001/in/S2B_MSIL1C_20181024T102059_N0206_R065_T33VWC_20181024T160131.SAFE/GRANULE/L1C_T33VWC_A008528_20181024T102428/IMG_DATA/T33VWC_20181024T102059_B05.jp2:Not a TIFF or MDI file, bad magic number 0 (0x0)  [vnsGDALLogInit.cxx:CPLIPFErrorHandler:59]
2019-05-10T09:18:51.797647 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] vnsCachingMacro Proceed caching... with file name vns_caching_TOAReader6.tif (ModeStreamDivisions=5; NumberOfStreamDivisions=800).  [vnsSentinel2L1ImageFileReaderBase.txx:GenerateTOACaching:490]

I have also tried looking into the code of start_maja.py and it seems that at some point no files are found for nomL2init_Natif and nomL2init_MUSCATE line 396-402), resulting in L2type to be set to None which does result in an error later (line 491-495) (I don't see any other place where L2type is defined in between).

        nomL2init_Natif = glob.glob("%s/%s" % (repL2, nomL2_par_dateImg_Natif[d]))
        nomL2init_MUSCATE = glob.glob("%s/%s" % (repL2, nomL2_par_dateImg_MUSCATE[d]))
        if len(nomL2init_Natif) > 0:
            derniereDate = d
            L2type = "Natif"

        elif len(nomL2init_MUSCATE) > 0:
            L2type = "MUSCATE"
            derniereDate = d

So then I tried to find out which folders it is searching in the above code and which folders are available and I found it is searching for: '../site_name/T33VWC/GIPP_001//S2?_OPER_SSC_L2VALD_T33VWC____20180601.DBL.DIR' (Natif) and ../site_name/T33VWC/GIPP_001//SENTINEL2?_20180601-*_TT33VWC_C_V*(MUSCATE) while only SENTINEL2A_20180601-102024-463_L2A_T33VWC_C_V1-0 was in this folder (I don't get where the second T in TT33VWC instead of T33VWC is coming from.

What do you think is the best approaching to debugging this issue?

I know this is all very complicated and explained poorly, but I hope there is a simple solution for all this. If you need any more information on my setup please ask!

olivierhagolle commented 5 years ago

Hi, my start MAJA is a little mess, that's not your fault, I am not a professional coder.

The GDAL error is not an error. That's something else Could you please provide the last lines of the MAJA processing logfile ? The command line could be useful, as well as the context (linux system...) ? Did you try the command line without T in the tile number ? Thanks, Olivier

olivierhagolle commented 5 years ago

Hi, I have updated start-maja to remove the T in the tile name provided in the command line, in case it is provided. I think it is the reason why it did not work. My test is still on going, so I am not sure it solves the issue Olivier

olivierhagolle commented 5 years ago

And you should consider working with MAJA 3.3 which was just released and improves a lot (particularly because a parameter issue we just found, see the plots at the end) http://www.cesbio.ups-tlse.fr/multitemp/?p=15666

daviddemeij commented 5 years ago

Hi Olivier,

Thanks for the fast response! I uploaded the log file (I hope it is the right one). S2A_MSIL1C_20180601T102021_N0206_R065_T33VWC_20180601T123308.SAFE.log

I am working on an Ubuntu 18.04.2 LTS server running the code in a tmux 2.6 terminal.

Maybe also useful information, these are all the files copied to my /tmp/in folder:

GIPP_S2_MAJA_3.3_TM                                                S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2A_MSIL1C_20180601T102021_N0206_R065_T33VWC_20180601T123308.SAFE  S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR         S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.HDR
S2A_MSIL1C_20180721T102021_N0206_R065_T33VWC_20180721T123132.SAFE  S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_MSIL1C_20180919T102021_N0206_R065_T33VWC_20180919T123234.SAFE  S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
S2A_MSIL1C_20181016T101021_N0206_R022_T33VWC_20181016T121930.SAFE  S2A_TEST_GIP_L2SMAC_L_ALLSITES_10005_20150703_21000101.EEF         S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_CKEXTL_S_31TJF____10001_20150703_21000101.EEF         S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF         S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR     S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL.DIR     S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR         S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.HDR         S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.DBL.DIR     S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.HDR         S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR         S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.DBL.DIR     S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR         S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR     S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.HDR
S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR         S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR         S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR     S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR         S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR     S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR     S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR         S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR         S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2COMM_L_ALLSITES_10009_20150703_21000101.EEF         S2B_MSIL1C_20180822T101019_N0206_R022_T33VWC_20180822T142412.SAFE  S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2COMM_L_ALLSITES_99999_20150703_21000101.EEF         S2B_MSIL1C_20180924T102019_N0206_R065_T33VWC_20180924T160501.SAFE  S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR     S2B_MSIL1C_20181014T102019_N0206_R065_T33VWC_20181014T140123.SAFE  S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR         S2B_MSIL1C_20181024T102059_N0206_R065_T33VWC_20181024T160131.SAFE  S2B_TEST_GIP_L2SMAC_L_ALLSITES_10005_20150703_21000101.EEF
S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.DBL.DIR     S2B_TEST_GIP_CKEXTL_S_31TJF____10001_20150703_21000101.EEF         S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.HDR         S2B_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF         S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR         S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.DBL.DIR     S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.HDR         S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR         S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR         S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR         S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.DBL.DIR     S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.HDR         S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2COMM_L_ALLSITES_10009_20150703_21000101.EEF         S2__TEST_AUX_REFDE2_T33VWC_0001.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR         S2B_TEST_GIP_L2COMM_L_ALLSITES_99999_20150703_21000101.EEF         S2__TEST_AUX_REFDE2_T33VWC_0001.HDR
S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR     S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR     S2__TEST_GIP_L2SITE_S_31TJF____10001_00000000_99999999.EEF
S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR         S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR

The output files I got are in this output folder: /home/ubuntu/efs/data/project/site_name/T33VWC/GIPP_001/ for example SENTINEL2A_20180601-102024-463_L2A_T33VWC_C_V1-0/SENTINEL2A_20180601-102024-463_L2A_T33VWC_C_V1-0_SRE_B2.tif.

My start_maja.py command was as follows: python2 ./start_maja.py -f folders.txt -g GIPP_001 -l LUT_MAJA_3_TM_CAMS -t T33VWC -s site_name -d 20180601 -e 20181101

daviddemeij commented 5 years ago

Thanks a lot I am going to try and will let you know if it works!

olivierhagolle commented 5 years ago

I inserted the parameters without CAMS yesterday: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_S2_MAJA_3.3_TM

Regarding the log file, which seems not to be terminated, I am not fully sure the tile number is the explanation, but still, start_maja was not working with a tile number starting with T. Now it is, at least in my test.

daviddemeij commented 5 years ago

It works now! So I guess it was because I added a "T" to the tile number (didn't realize that T wasn't part of the tile number).

I was just wondering if the resulting product already takes solar irradiance into account or if we should still normalize for this.

olivierhagolle commented 5 years ago

Good news ! We provide surface reflectances, which are independent and corrected from solar irradiance. The only thing which is not corrected are the directional effects. We do not normalise to nadir viewing direction, and keep the observation sun angles.

Remember, you should swap to V3.3 if not done yet. I am closing the issue, but you may reopen it or a new one if needed.

daviddemeij commented 5 years ago

Okay, good to know!

For some reason some images are still a lot brighter than others, I thought this would be more normalized over-time in this process. I haven't used the CAMS data yet though, maybe this would help fix this issue.

I did swap to V3.3 already.

olivierhagolle commented 5 years ago

Hi David, That's strange, we usually do not observe that. Some images may be slightly brighter but usually not a lot brighter. Did you compare the surface reflectance values ? Or just looked at the images ? Did you set a constant min and max reflectance values before displaying the images ? Sometimes, the automatic display adapts its min and max values to the image content, which makes time series inconsistent.

That's a common error among beginner users (but maybe are you an expert one).

If my guess of the cause of what you observe is wrong, I am interested to have the dates of the images you used.

best regards, Olvier

daviddemeij commented 5 years ago

Hi Olivier,

Thanks for your response. I am not an expert, but I did clip the images at the same maximum and minimum value and I performed min-max normalization using the same min and max values for each image. So I think something else must be going wrong. I am now running the code using the CAMS data, so hopefully, this will result in more equal images over time.

I ran the code on the following images:

S2A_MSIL1C_20180601T102021_N0206_R065_T33VWC_20180601T123308.SAFE                                                                                                         
S2A_MSIL1C_20180721T102021_N0206_R065_T33VWC_20180721T123132.SAFE                                                                                                                    
S2A_MSIL1C_20180919T102021_N0206_R065_T33VWC_20180919T123234.SAFE                                                                                                         
S2A_MSIL1C_20181016T101021_N0206_R022_T33VWC_20181016T121930.SAFE                                                                                                                    
S2B_MSIL1C_20180507T102019_N0206_R065_T33VWC_20180507T123037.SAFE                                                                                                         
S2B_MSIL1C_20180822T101019_N0206_R022_T33VWC_20180822T142412.SAFE                                                                                                                    
S2B_MSIL1C_20180924T102019_N0206_R065_T33VWC_20180924T160501.SAFE                                                                                                         
S2B_MSIL1C_20181014T102019_N0206_R065_T33VWC_20181014T140123.SAFE                                                                                                                    
S2B_MSIL1C_20181024T102059_N0206_R065_T33VWC_20181024T160131.SAFE

Where it turned out 2018-08-22 where is very bright (and also quite cloudy), also 2018-07-21 is very bright and 2018-10-16 has some blue artifacts (due to clouds probably). The other images do seem to be normalized nicely.

I added some cropped images: 20180601 20180721 20180821 20180919 20180924 20181014 20181024

Zoomed out 2018-08-22: 20180822_full

daviddemeij commented 5 years ago

Also when dividing the images by 10,000 the maximum value I saw was around 1.6, I guess the values are supposed to be within 0 and 1?

Update: Here are the statistics for all images for each band.

SENTINEL2A_20180601-102024-463_L2A_T33VWC_C_V1-0
band    min max mean    median
FRE_B2  0.0 1.6650  0.0178  0.0154
FRE_B3  0.0 1.6559  0.0280  0.0224
FRE_B4  0.0 1.5661  0.0201  0.0129
FRE_B5  0.0 1.3713  0.0433  0.0355
FRE_B6  0.0 1.3318  0.1188  0.1229
FRE_B7  0.0 1.3482  0.1439  0.1483
FRE_B8  0.0 1.3722  0.1486  0.1509
FRE_B8A 0.0 1.3235  0.1588  0.1670
FRE_B11 0.0 1.2384  0.0855  0.0718
FRE_B12 0.0 1.1971  0.0456  0.0313
SENTINEL2A_20180721-102024-461_L2A_T33VWC_C_V1-0
band    min max mean    median
FRE_B2  0.0 1.7459  0.0257  0.0227
FRE_B3  0.0 1.6291  0.0358  0.0286
FRE_B4  0.0 1.5343  0.0288  0.0137
FRE_B5  0.0 1.4838  0.0470  0.0304
FRE_B6  0.0 1.4339  0.1083  0.1193
FRE_B7  0.0 1.4103  0.1312  0.1465
FRE_B8  0.0 1.4319  0.1361  0.1480
FRE_B8A 0.0 1.3809  0.1482  0.1664
FRE_B11 0.0 1.2933  0.0942  0.0679
FRE_B12 0.0 1.2354  0.0548  0.0319
SENTINEL2A_20180919-102018-462_L2A_T33VWC_C_V1-0
band    min max mean    median
FRE_B2  0.0 1.8974  0.0290  0.0076
FRE_B3  0.0 1.8209  0.0344  0.0101
FRE_B4  0.0 1.7835  0.0304  0.0011
FRE_B5  0.0 1.7411  0.0475  0.0182
FRE_B6  0.0 1.6652  0.1050  0.0882
FRE_B7  0.0 1.6262  0.1243  0.1097
FRE_B8  0.0 1.6013  0.1280  0.1066
FRE_B8A 0.0 1.5760  0.1394  0.1271
FRE_B11 0.0 1.3292  0.0883  0.0473
FRE_B12 0.0 1.2670  0.0537  0.0200
SENTINEL2A_20181016-101021-456_L2A_T33VWC_C_V1-0
band    min max mean    median
FRE_B2  0.0 2.0824  0.0126  0.0097
FRE_B3  0.0 1.8936  0.0175  0.0094
FRE_B4  0.0 1.7662  0.0144  0.0000
FRE_B5  0.0 1.7167  0.0334  0.0152
FRE_B6  0.0 1.6544  0.0819  0.0742
FRE_B7  0.0 1.6203  0.0979  0.0907
FRE_B8  0.0 1.6231  0.1033  0.0822
FRE_B8A 0.0 1.5738  0.1153  0.1088
FRE_B11 0.0 1.4039  0.0674  0.0337
FRE_B12 0.0 1.3233  0.0339  0.0100
SENTINEL2B_20180822-101321-460_L2A_T33VWC_C_V1-0
band    min max mean    median
FRE_B2  0.0 1.4622  0.0703  0.0115
FRE_B3  0.0 1.4914  0.0771  0.0194
FRE_B4  0.0 1.4641  0.0762  0.0086
FRE_B5  0.0 1.4348  0.0904  0.0371
FRE_B6  0.0 1.4174  0.1388  0.1270
FRE_B7  0.0 1.3969  0.1564  0.1516
FRE_B8  0.0 1.3828  0.1617  0.1566
FRE_B8A 0.0 1.3773  0.1722  0.1748
FRE_B11 0.0 1.2961  0.1339  0.0840
FRE_B12 0.0 1.2404  0.0992  0.0378
SENTINEL2B_20180924-102016-460_L2A_T33VWC_C_V1-0
band    min max mean    median
FRE_B2  0.0 1.6371  0.0156  0.0109
FRE_B3  0.0 1.4923  0.0227  0.0158
FRE_B4  0.0 1.3964  0.0178  0.0087
FRE_B5  0.0 1.3150  0.0360  0.0301
FRE_B6  0.0 1.2725  0.0916  0.1035
FRE_B7  0.0 1.2578  0.1097  0.1252
FRE_B8  0.0 1.3172  0.1165  0.1271
FRE_B8A 0.0 1.2431  0.1251  0.1448
FRE_B11 0.0 1.1922  0.0712  0.0485
FRE_B12 0.0 1.2158  0.0400  0.0206
SENTINEL2B_20181014-102019-459_L2A_T33VWC_C_V1-0
band    min max mean    median
FRE_B2  0.0 1.8357  0.0136  0.0107
FRE_B3  0.0 1.7244  0.0205  0.0135
FRE_B4  0.0 1.6244  0.0162  0.0039
FRE_B5  0.0 1.5809  0.0351  0.0252
FRE_B6  0.0 1.5301  0.0830  0.0895
FRE_B7  0.0 1.5078  0.0989  0.1090
FRE_B8  0.0 1.5392  0.1062  0.1084
FRE_B8A 0.0 1.4656  0.1166  0.1293
FRE_B11 0.0 1.3381  0.0667  0.0387
FRE_B12 0.0 1.3069  0.0347  0.0139
SENTINEL2B_20181024-102428-105_L2A_T33VWC_C_V1-0
band    min max mean    median
FRE_B2  0.0 1.6393  0.0177  0.0152
FRE_B3  0.0 1.4893  0.0259  0.0197
FRE_B4  0.0 1.3889  0.0226  0.0130
FRE_B5  0.0 1.3685  0.0404  0.0325
FRE_B6  0.0 1.2756  0.0815  0.0881
FRE_B7  0.0 1.2624  0.0950  0.1047
FRE_B8  0.0 1.3302  0.1020  0.1040
FRE_B8A 0.0 1.2423  0.1107  0.1238
FRE_B11 0.0 1.2788  0.0663  0.0424
FRE_B12 0.0 1.2338  0.0377  0.0184
olivierhagolle commented 5 years ago

Hi, that's interesting...

I doubt the use of CAMS can solve that, usually the aerosol amount in your regions is low, and CAMS just brings more accuracy on the aerosol type which has little influence if the aerosol content is low.

I am going to process these images on my side in order to check, because I have never seen such variations. I have started the processing with all dates between May and October. I'll have the results tomorrow morning. I'll try to analyse that before the end of the week.

Reflectance values can be greater than 1, it is the albedo which cannot be greater than 1 (the albedo if the integration of reflectance on the half space). For instance, a mirror that reflects the sun towards the satellite can have a reflectance of 1000. Or a slope in a cloud surface facing the sun can also reach 1.3 - 1.5. 2 is a lot, except in the case of a mirror (roof top ass, green house, car glasses...)

I'll let you know about my results. Olivier

olivierhagolle commented 5 years ago

Hi, I have had a look at the images I generated using Start_MAJA with version 3.3 without CAMS, and I see nothing wrong. Here is a little animation, at a low resolution because of the size of data. But you will see that everything is quite consistent. I have overlayed the contours of cloud mask (green) shadows mask (yellow), water mask (blue). You can see that the image of 20180822, the last one is correct, and surface has reflectances close to those of the other dates. Same for 20180721. The island (or peninsula) in the east becomes slowly brighter but I guess it is because of summer.

My quicklooks use, for all dates: min=0 max=2500 (or 0.25 if you convert to reflectance floats first).

T33VWC

So it still looks as if there was an issue in your quicklook generation.

Just a question though, did you download all the date, or only the ones you quoted in your email. MAJA works better if all the dates are provided.

Best regards, Olivier

daviddemeij commented 5 years ago

Hi Olivier,

Thanks very much for uploading this, this will be very helpful for debugging. I did not download all the dates, just the dates I mentioned (I tried filtering out dates with too many clouds and dates that were very close to each other), but I will try downloading all the dates in the period now. I think I might have used the CAMS parameters while not using CAMS data, maybe this could have resulted in some problems?

daviddemeij commented 5 years ago

It seems to be working now! I think it was an issue with using the wrong parameters.

olivierhagolle commented 5 years ago

Good news, thanks ! Olivier