CDS beta raises unclear error about accepting licenses

veenstrajelmer commented 1 month ago

What happened?

I am trying to update my current workflow to how it is described in https://confluence.ecmwf.int/display/CKB/Please+read%3A+CDS+and+ADS+migrating+to+new+infrastructure%3A+Common+Data+Store+%28CDS%29+Engine However, I am encountering several issues that require fixes.

What are the steps to reproduce the bug?

Update ~.cdsapirc according to specifications on https://cds-beta.climate.copernicus.eu/how-to-api Run simple code:

import os
import cdsapi

# environment variables take precedence over the ~/.cdsapirc file, so useful for testing
os.environ["CDSAPI_URL"] = "https://cds-beta.climate.copernicus.eu/api"
os.environ["CDSAPI_KEY"] = "[insert your apikey here]"

request_dict = {'product_type':'reanalysis',
                'variable':'msl',
                'year':'2010',
                'month':'01',
                'day':'01',
                'time':'00:00',
                'area':[52, 2, 51, 3], # north, west, south, east.
                'format':'netcdf'}

c = cdsapi.Client()
c.retrieve(name='reanalysis-era5-single-levels', request=request_dict)

This raises "RuntimeError: 403 Client Error: required licences not accepted required licences not accepted; please accept the following licences to proceed: [{'id': 'licence-to-use-copernicus-products', 'revision': 12}]"

However, I have already accepted all licenses on the https://cds-beta.climate.copernicus.eu/profile page, so this seems to be an incorrect error.

Addtional requests

I noticed that the new url contains the word "beta". I expect this will be removed again in the new future when the "new CDS" is not beta anymore? It would be inconvenient if users have to update their ~/.cdsapirc file again when this happens. Would it be possible to choose a url that will not change in the near future? Maybe including a version number like before?
would it be possible to promt for the users apikey instead of letting the user add their file manually? I know it is also possible to provide credentials via environment variables, this is already helpful, but a cdsapi.login() function would also be convenient.

Version

0.7.0

Platform (OS and architecture)

Windows 10 64 bit

Relevant log output

No response

Accompanying data

No response

Organisation

Deltares

negedng commented 1 month ago

Hi,

I think this is a dataset-specific term, not the general one you accept when logging in.

What worked for me was going to the dataset page manually, accepting the terms, and running the API script again.

I did it for this ERA5 dataset: https://cds-beta.climate.copernicus.eu/datasets/reanalysis-era5-land?tab=download

veenstrajelmer commented 1 month ago

I expected something like this. However, this is not completely clear from the error message. It would be convenient if the error would provide a URL where to accept these terms. Furthermore, I think it would be more convenient to centralize the acceptance of terms in the users account page (not only keep track of them there once accepted).

I just accepted these dataset-specific terms, but would now like to simulate having not accepted them. Since all users of the toolbox that I maintain have not yet accepted them. I cannot un-accept the license though. This would have been no issue if the error message would have been clear to start with, but now I cannot reproduce a confusing error message to avoid issues for other users.

Would it be possible to pick up one of the suggested solutions?

hannahker commented 1 month ago

Seconding this as a paint point when getting started with the beta API! It took me a while to figure out how to accept the correct license, so a link to the license in the error message would have been very useful.

ALopAlos commented 1 month ago

Thanks a lot for your comments, Recommended good practice is first to try a request of the desired dataset via de interactive form and then use that request on "show API request" as a template when using the API. This way it is guaranteed that Licence is accepted. About the error, the licence is there: licence-to-use-copernicus-products . Do you consider this is not informative/clear enough? In case, what info are you missing? Thanks!

veenstrajelmer commented 1 month ago

I know that is your ideal situation, but we are working on automating model generation workflows for others, to prevent each user from undertaking that manual step.

Either way, as both @hannahker and myself already suggested, it would be great to have a URL in the error message that points to the dataset page where the user can accept the license. Alternatively, just add the license to the user profile page (also before it is accepted), so the user can accept all required licenses at once.

negedng commented 1 month ago

I used a 3rd party script to download it, it told me to register, but I don't even know which ERA5 variation I tried to download. Luckily all have the same terms. A link to the dataset page would be nice

ALopAlos commented 1 month ago

Thanks, your suggestion are very welcome. A related story has been included on the CDS-Beta project backlog to address this topic.

thopri commented 2 weeks ago

I too have found this confusing, there was no clear way to accept the licence and the api requests were ones I had made previously on an older version of the API so I would have expected to have already accepted them? Having a URL to visit in the error would be very helpful. In the end i had to figure out what dataset I was actually downloading (I am using an automated model configuration generator) and then make an random request on its web form where the licence was somehow accepted? and my API requests started working.

daniloceano commented 1 week ago

I too have came across this error and wasn't able to find a solution until I came up with this page. If I may: I believe that displaying the link for accepting the license in the error message would be the most user-friendly solution. If that's not a desirable solution, the error message could be more informative as I had no clue what to do as I was certain that it was referring to the cdsapi license

ecmwf / cdsapi