search

Evoland-Land-Monitoring-Evolution / sentinel2_superresolution

Super-resolution of 10 Sentinel-2 bands to 5-meter resolution, starting from L1C or L2A (Theia format) products.
Apache License 2.0
52 stars 7 forks source link

Very high ram usage for bicubic upsampling option #9

Open ThomasCusson opened 2 months ago

ThomasCusson commented 2 months ago

Hello,

I ran the process on a full S2 tile for about 1h30. All went well, until the progress bar was full, then the process started using a lot of ram, above 100GB, so it crashed because of the lack of memory on my machine.

I relaunched without the "--bicubic" option, and all went well. It seems that it is because it upscales the whole image at once instead of using chunks, filling up the memory when handling large images.

jmichel-otb commented 2 months ago

Thank you for reporting this. Never add the issue, but I guess I only ran extracts on my laptop and full product on HPC nodes where RAM is not that limited. I will add chunking in bicubic mode also.

jmichel-otb commented 2 months ago

Fix pushed here : https://github.com/Evoland-Land-Monitoring-Evolution/sentinel2_superresolution/commit/49652ab346c4704eb88a3bf482c17c7c00aa46ff

Can you confirm it works on your side ?

jmichel-otb commented 2 months ago

It did not remove the model, it moved it to another folder so that it can be installed by pip. You can find it in src/sentinel2_superresolution/models/carn_3x3x64g4sw_bootstrap.onnx.

Alternatively, if you re-install following the latest directions, it should work without providing the model to the command-line:

$ pip install git+https://framagit.org/jmichel-otb/sentinel2_superresolution.git

or $ pip install "sentinel2_superresolution[gpu] @ git+https://github.com/Evoland-Land-Monitoring-Evolution/sentinel2_superresolution.git"

(from updated README)

ThomasCusson commented 2 months ago

Yes, sorry I wasn't paying attention, I removed my previous message. I'm currently re-running it with the "--bicubic" option, I'll notice you if it worked.