search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

Disk space, Processing #30

Closed olivierhagolle closed 4 years ago

olivierhagolle commented 4 years ago

Hello Olivier,

I hope you are enjoying your vacation. Gonzalo and I have had a chat to update me what happened during my absence. MAJA is still causing several issues for us.

One of them is the processing time and the space required. I tried to run it on one month's worth of data (9 images) and we provided around 30GB of disc space, which wasn't sufficient. It produced an error after over 1 hour running time due to lack of space. Could you give us an estimation how much disk space and time is needed for processing one tile? Alternatively, is there a way to reduce the disk space needed for the processing?

Secondly, since we want to investigate a larger region, we aim at reusing the flat surface reflectances so we don't have to go through the entire multi-temporal processing chain every time a new image for a tile comes in. Could you please give us an idea how we can reuse it and how it will affect the processing parameters (disk space/ time)?

Thanks for your help! Daria

Originally posted by @DariaLudtke in https://github.com/CNES/Start-MAJA/issues/24#issuecomment-511345215

olivierhagolle commented 4 years ago

Hi Daria, I just saw this request among the replies to another one, I am sorry I have missed it in the first place. I have created a new issue so that different subjects don't get mixed.

Each L1C product takes around (800 GB) Each L2A product (uncompressed) takes 2.8 GB, but the volumes would be much reduced after compression). You can also remove either the SRE or FRE file if you are sure you do not need them.

As a result, 30GB is clearly not enough. You should consider 300...

For a given time series, you do not need to process the whole multi-temporal processing every time there is a new date. Start_MAJA checks for the existence of a previous L2A product, and only processes the new ones. But this doesn't work if you have compressed or moved the L2A products already generated. You should at least keep the last one in your processing environment.

Hope it answers your questions. Sorry again for the delays in answering Olivier

olivierhagolle commented 4 years ago

Closed, no reply for >1 month. feel free to reopen.

Seam8 commented 4 years ago

Hello,

A question about processing : Is it possible to reduce the region of processing ? Can we run MAJA on an area smaller than a whole Sentinel tile ? By clipping L1C maybe but SAFE format would risk to become invalid.

Thank you, Selim

olivierhagolle commented 4 years ago

Hi, yes that would be complex, and with MAJA current version, you would not gain much processing time. We intend to work on that with MAJA V4 Olivier