Impossible to detect a L1 product in the directory </data/input>.

[j08lue]

Hi @olivierhagolle, thanks for pushing for wider usage of MAJA. I just saw here that we should raise issues regarding MAJA here on this repo. I also already wrote to maja-support@cnes.fr, please disregard that mail if it also ends up on your desk.

I am trying out MAJA for atmospheric correction and cloud masking, but have some issues regarding selection of Sentinel 2 L1C products.

I was able to install and run MAJA based on the instructions in https://github.com/olivierhagolle/Start_maja with a somewhat modified Dockerfile (I'll contribute back once I have a fully working setup).

I am running MAJA from command line with commands like

docker run -v /share/maja/32UNG:/data maja /opt/maja/core/1.0/bin/maja -i /data/input -o /data/output -m L2NOMINAL -ucs /data/userconf --TileId 32UNG

In the /data/input folder, I place everything:

• the DTM data prepared with scripts from http://tully.ups-tlse.fr/olivier/prepare_mnt,
• a MAJA Sentinel 2 L2A product I produced with in L2BACKWARD mode, and
• a single Sentinel 2 L1C product.

Some L1C products are processed fine and I get the intended MAJA output. But sometimes, for no obvious reason, I get the error

Impossible to detected a L1 product in the directory </data/input>.

even though the extracted .SAFE product in my /data/input folder is structured in exactly the same way as the other products.

Can you help me find out why some of the L1C products are not detected? How does MAJA search for them? Obviously not just by glob(/data/input/S2?_OPER_PRD_*.SAFE)?

By the way, the Sentinel 2 products I am trying to process are from before 2016-11, so in the clumsy old full-scene multi-tile format. But all of them are - both the working and not-working ones.

[olivierhagolle]

Hi Jonas, with the information provided, I do not see the cause of the issue.

Before asking you for more details, did you try to use directly the start_maja.py python script ? Because it should handle properly the execution of MAJA for a whole time series, starting from the backward mode. It also handles seamlessly the old S2A format as well as the new one.

Thanks, Olivier

[j08lue]

@olivierhagolle Thanks for getting back so quickly!

did you try to use directly the start_maja.py python script?

Yes/no, I had a hard time figuring out the folder structure that the script expects, and what it does with it.

But I think after some hours with MAJA I now better understand it: You index the input products by date and then create symlinks to the ones you need in the input directory.

But I see now that the symlinks you create are actually the GRANULE folders, not the full .SAFE products, is that correct? E.g. here you create the product list:

https://github.com/olivierhagolle/Start_maja/blob/a6e6c4d260b837e113f4610a8d9ab53b6f7e42bc/start_maja.py#L151-L152

and then index them by date:

https://github.com/olivierhagolle/Start_maja/blob/a6e6c4d260b837e113f4610a8d9ab53b6f7e42bc/start_maja.py#L201

and finally create the symlinks:

https://github.com/olivierhagolle/Start_maja/blob/a6e6c4d260b837e113f4610a8d9ab53b6f7e42bc/start_maja.py#L242

So the MAJA input directory should not contain full .SAFE products, but only the GRANULE folders?

[j08lue]

So the MAJA input directory should not contain full .SAFE products, but only the GRANULE folders?

I just tried to put only the granule folder into the input directory and still get the same error (no L1 products detected).

[olivierhagolle]

Start_maja.py was meant to be simple, so it seems I missed my target. What information would you have needed ?

Yes your understanding of start_maja is good, but in fact, the old and new format have globally the same structure, except that there is only one granule in GRANULES folder. So I do not think it will solve the issue.

[olivierhagolle]

Well it didn't ;)

[olivierhagolle]

Can I do a try with start_maja ? What is the tile number and time period ?

[j08lue]

Thanks, please: Tile is 32UNG and period is 2015-08-09 to 2015-10-01 (only scenes with cloud cover under say 50%).

[olivierhagolle]

Ok, I'll need some time, to download the files (not working on AWS...) By the way, MAJA will work better if you feed it with all products in the time series, not just the ones with low cloud cover. Olivier

[j08lue]

By the way, MAJA will work better if you feed it with all products in the time series, not just the ones with low cloud cover.

Thanks, that is valuable information. I have been wondering about that. I'd like to hear more about the reasons behind that, maybe better in an offline discussion somewhere else.

[j08lue]

Start_maja.py was meant to be simple, so it seems I missed my target. What information would you have needed ?

No/yes, the script became easier to understand once I understood what final folder structure MAJA expects.

An annotated structure (aka arborescence) in the manual ([MU] SETG-MU-MAJA-010-CS.pdf) would be good, I think. The important part is what MAJA expects to find in the input folder and how it looks for it.

The script would benefit from more expressive variable names (fewer abbreviations), os.path.join(one, two, three) instead of '%s/%s/%s'%(one,two,three), general PEP8 style, Python-3-compatible print(), and more comments. But that is really mostly a matter of taste. 😉

[olivierhagolle]

The main criteria within MAJA suppose that surface reflectance does not change between two successive acquisitions, to detect clouds and estimate aerosols. So the closer the acquisitions, the better. Of course, in this context, cloud free images bring much more information, but an image with 25% cloud free pixel can still be valuable.

[olivierhagolle]

Thanks for changing the title (even if the typo originates from MAJA V1_0 messages, but corrected afterwards).

I ran the same tiles from 32UNG in 2015 and... reproducted the errors for three products. S2A_OPER_PRD_MSIL1C_PDMC_20161214T202954_R008_V20151021T104102_20151021T104102.SAFE S2A_OPER_PRD_MSIL1C_PDMC_20161231T080544_R108_V20150928T103036_20150928T103036.SAFE. S2A_OPER_PRD_MSIL1C_PDMC_20170122T053150_R108_V20150829T103026_20150829T103026.SAFE

So, looking for the reason why is on our side now. Thanks for the bug report. Our bug correction is rather slow, do no expect a quick answer ;) Best regards, Olivier

[j08lue]

@olivierhagolle Thanks a lot for digging into this, too. I'll follow the evolution of your system. Please keep me posted.