search

sepal-contrib / sdg_15.3.1

monitor SDG indicators at plot level
MIT License
3 stars 3 forks source link

download hangs at "merging tile from drive" #89

Closed i-m-amit closed 2 years ago

i-m-amit commented 2 years ago

There is an issue with the download function, it takes random files from the drive to the sepal folder and never ends the process.

There is a typo at https://github.com/12rambau/sdg_indicators_module/blob/e8f8ac07638266623cc05264998c656848303e43/component/scripts/run_15_3_1.py#L48

12rambau commented 2 years ago

I tried from my side on Singapore as usual and I didn't manage to reproduce your issue. Could you please send me more details on what are the parameters you used ?

Also if you have images that starts with the same name as the one we are building then it will effectively crash.

i-m-amit commented 2 years ago

I have tried with various parameters and the result is always same! here are the outputs of two results:

Screenshot 2021-11-10 at 1 16 43 PM

These random files are located in subdirectories and shared folder. fig2...png is available inside deccma>MD_Chapter_3>fig and all the land cover are in a sub directories of a shared folder shared by Rashed.

i-m-amit commented 2 years ago

Instead of searching the entire drive, can we create an unique folder to download the results and then run the re in that particular folder only?

12rambau commented 2 years ago

the function is widely used in all python workflows, until I found the time to code the task-manager app in SEPAL I would prefer to not change it.

Instead we are going to use very detailed file names that cannot be confused (and we don't really care as in the end these files are flushed from Gdrive anyway)

I just need to make some tests the naming convention of GEE is very restrictive

12rambau commented 2 years ago

I've setup more sophisticated prefix, I'll let you try the master branch from your side and if it's working I'll push it to release

i-m-amit commented 2 years ago

what about changing the prefix to sepal_sdg_15_3_1 and adding a condition in the query string like fullText contains 'sepal_sdg_15_3_1' to filter only those contain this prefix?

12rambau commented 2 years ago

what about changing the prefix to sepal_sdg_15_3_1

Let's assume the application freezes and the files stay in your drive, I need to make sure that another launch with different parameters will not retreive these files

fullText contains 'sepal_sdg_15_3_1'

Here it's even more dangerous as it will download all the indicators every time

Did you had the chance to test the last implementation ?

i-m-amit commented 2 years ago

what about changing the prefix to sepal_sdg_15_3_1

Let's assume the application freezes and the files stay in your drive, I need to make sure that another launch with different parameters will not retreive these files Is it possible to delete those files before executing the export task?

fullText contains 'sepal_sdg_15_3_1'

Here it's even more dangerous as it will download all the indicators every time

Did you had the chance to test the last implementation ?

Yes, now I am getting an error that says : NoneType object has no attributes 'state'

12rambau commented 2 years ago

that was linked to naming errors. closing as it seems to work now.