sepal-contrib / sdg_15.3.1

monitor SDG indicators at plot level
MIT License
3 stars 3 forks source link

user memory limit exceeded error #54

Closed yfinegold closed 2 years ago

yfinegold commented 3 years ago

There is an error when running the app for different AOIs

I tried with American Samoa - a tiny island country and got this error after some minutes. Error message (seems to be from GEE): 'User memory limit exceeded.'

Parameters: AOI: American Samoa Start history: 2015 End history: 2017 Start monitoring: 2018 End monitoring: 2020 Select sensor: all sensors Select trajectory: ndvi trend transition matrix: default climate regime options: per pixel based on global climate data

I also tried with an admin 2 boundaries and the same parameters and got this error: 'Computation timed out.'

12rambau commented 3 years ago

@i-m-amit as a starting point for your debugging step you can have a look here: https://developers.google.com/earth-engine/guides/debugging#user-memory-limit-exceeded

i-m-amit commented 3 years ago

@12rambau The for-loop based Mann Kandal function is efficient than the new server-side approach!

12rambau commented 3 years ago

damned (that's very... unexpected). Then let me know when you have something that works so that I can push it to SEPAL servers

i-m-amit commented 3 years ago

The main bottleneck here is the Kendall stats. Even adapting to the built-in method does not provide the desired result. So, I have tested nearly all the functions to locate the exact problem as the GEE error message is useless!. The tool is working with the given AOI until I combined the Landsat and Sentinel data. This might be related to the reprojection of the Landsats in S2's resolution!. I have got the same error after changing the resolution of S2 to resulution of L8. Mann_kendall_efficiency_test.ipynb.zip All the three Kendal functions in this notebook fail to produce an output if I combine Landsats and S2 sensors. The built-in method works only with the getInfo() call.

12rambau commented 3 years ago

does it works if we use only S2 data ?

i-m-amit commented 3 years ago

Yes, It does.

12rambau commented 3 years ago

then what do you think if I prevent the usage of a combination of sentinel 2 and landsat ?

I mean forcing the user to choose between the 2 sat families

i-m-amit commented 3 years ago

Yes, It is a logical solution. Exporting the annual ndvi series as an asset and then reuse in the subsequent processes could also be a solution.

12rambau commented 2 years ago

exporting is a super long process we should always try to avoid it (and we obviously cannot create a worldwide ndvi dataset)

12rambau commented 2 years ago

Capture d’écran 2021-07-05 à 10 15 19

Seems to work after your modifications and after preventing the mix of sentinel and landsat. @yfinegold I'm going to push to prod and let you test it. Close or comment further this issue if you are still experiencing problems

yfinegold commented 2 years ago

I am able to run and produce the productivity and indicator maps now. But no longer able to download any of the results. The productivity and indicator 15.3.1 results look exactly the same on the map to me - is that expected?

12rambau commented 2 years ago

I'm correcting the bug for the download (it was related to the jump from sepal_ui v1 to v2.

looking at the American samosa' results it appears that:

As the final indicator is a combination of these 3 (winner gets all) productivity is the main drive for the final results. You can still see the influence of the SOC layer in the artefacts near the coast (red vertical and horizontal lines)