Issue during the download process

[CharlesNickmilder]

Hello, I tried to use the script to download Sentinel 2 L2A data. I tried both demo comand line and a more recent one on our Linux server given that it does not work on windows: (1)python ./theia_download.py -t T31TCJ -c SENTINEL2 -a config_theia.cfg -d 2016-09-01 -f 2016-10-01 (2)python ./theia_download.py -t T31UFR -c SENTINEL2 -a config_theia.cfg -d 2020-09-01 -f 2020-09-15 Here is the full comand line and answers:

python ./theia_download.py -t T31UFR -c SENTINEL2 -a config_theia.cfg -d 2020-09-01 -f 2020-09-15 Get theia single sign on token Done https://theia.cnes.fr/atdistrib/resto2/api/collections/SENTINEL2/search.json?completionDate=2020-09-15&startDate=2020-09-01&maxRecords=500&location=T31UFR&processingLevel=LEVEL2A curl -k -o search.json "https://theia.cnes.fr/atdistrib/resto2/api/collections/SENTINEL2/search.json?completionDate=2020-09-15&startDate=2020-09-01&maxRecords=500&location=T31UFR&processingLevel=LEVEL2A" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 28213 0 28213 0 0 64266 0 --:--:-- --:--:-- --:--:-- 64266

(u'SENTINEL2A_20200914-104713-474_L2A_T31UFR_D', u'42813639-39ba-537d-a296-672a1c9dd728') ('cloudCover:', 4) ('acq date', u'2020-09-14T10:', 'prod date', u'2020-09-21T11:', 'pub date', u'2020-09-21T13:') ('>>>>>>>', u'SENTINEL2A_20200914-104713-474_L2A_T31UFR') curl -o "./SENTINEL2A_20200914-104713-474_L2A_T31UFR_D.tmp" -k -H "Authorization: Bearer 27dd70a0-cc19-3ee9-862a-fbbcea71005d" https://theia.cnes.fr/atdistrib/resto2/collections/SENTINEL2/42813639-39ba-537d-a296-672a1c9dd728/download/?issuerId=theia % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 103 100 103 0 0 121 0 --:--:-- --:--:-- --:--:-- 120 Result is a text file {u'ErrorCode': 403, u'ErrorMessage': u'User profile has not been validated. Please contact an administrator'}

Thanks to the very explicit messages, it appears that I managed to get an text file informing me that I had no access to the data although my account was validated and I can edit it on https://sso.theia-land.fr/. Unlike #20 I can't download a single product whether on catalogue.theia-land.fr or via your code but it works well via https://theia.cnes.fr/. Could you please help me?

[olivierhagolle]

Hi, I have just tried the same commands with my account and it works. I obtain the same message as the one you observe if my account and password are not well specified in the config_theia.cfg file.

Did you do it properly ? Does this account enables you to download data from https://theia.cnes.fr/. One possibility is that the your password contains special caracter that my code does not handle well. Then, a password with only alphanumeric could solve the issue.

Best regards, Olivier

[CharlesNickmilder]

Hi, Thanks for the answer. I manage to download directly from https://theia.cnes.fr/ and through aria2 and meta4 files generated on this website.

Concerning the specification of the user account and the password, here is what I use:

serveur = https://theia.cnes.fr/atdistrib resto = resto2 token_type = text login_theia = mymail@uliege.be password_theia = mypassword Should there quotation marks or end of line markers?

Concerning the password handling. What are the characters not handled correctly? I tried an alphanumeric password without special characters but containing upper and lower case and another only lower case and numbers with a dot (.) character. Both failed. Given how the code was written, the password seems to be stored in a python object. I don't think python is case-sensitive, could it be a clue to find the source of the mishandling?

Regards, Charles

[olivierhagolle]

Hi Charles, the file looks ok, there is no need for end of line markers. Python is case sensitive, so it does not seem to be the issue. The password with only alphanumeric caracters should have worked. I am sorry, I have no clue.

The EODAG tool has an interface to Theia, and many others, and is much more professional code. You might give it a try. https://eodag.readthedocs.io/en/latest/# https://github.com/CS-SI/eodag

Olivier

[fwrite]

I get a very similar error, but only after three downloads success, i.e. the fourth download fails. I have repeatedly been in touch with Theia support about the issue, as it appears to be on their side. Could it be throttling that does not release old connections fast enough? I could image that your limits are more relaxed, @olivierhagolle?

[olivierhagolle]

Sorry for that. That's strange, but such an issue is probably not on theia-download side. Please keep insisting with Theia, and tell me if you need that I add some pressure so that your issue is solved.

[chiararik]

Hello,

I am currently facing the same issue:

C:\Users\Chiara>python C:/Tools/theia_download/theia_download.py -t T32TMS -c Snow -a C:/Tools/theia_download/config_theia.cfg -d 2016-01-01 -f 2016-12-31
Get theia single sign on token
Done
https://theia.cnes.fr/atdistrib/resto2/api/collections/Snow/search.json?location=T32TMS&startDate=2016-01-01&completionDate=2016-12-31&maxRecords=500
curl -k  -o search.json "https://theia.cnes.fr/atdistrib/resto2/api/collections/Snow/search.json?location=T32TMS&startDate=2016-01-01&completionDate=2016-12-31&maxRecords=500"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1065  100  1065    0     0   1065      0  0:00:01 --:--:--  0:00:01  1420

C:\Users\Chiara>

it is very strange because until a few months ago it worked!

Has anyone managed to fix the problem?

Thanks, best regards, Chiara

[fwrite]

C:\Users\Chiara>python C:/Tools/theia_download/theia_download.py -t T32TMS -c Snow -a C:/Tools/theia_download/config_theia.cfg -d 2016-01-01 -f 2016-12-31 [...] https://theia.cnes.fr/atdistrib/resto2/api/collections/Snow/search.json?location=T32TMS&startDate=2016-01-01&completionDate=2016-12-31&maxRecords=500

If you open the URL printed by theia_download.py in your browser (shown above), it will list zero available tiles. Could it be that this region is (no longer) available with Theia?

[chiararik]

Yes, you are right! there is no download issue, research itself was the problem. Thank you!