search

ecmwf / climetlab

Python package for easy access to weather and climate data
Apache License 2.0
374 stars 57 forks source link

INFO Request is queued #28

Closed gewitterblitz closed 3 years ago

gewitterblitz commented 3 years ago

I am trying to access ERA5 data using both cdsapi and climetlab. Just replicating the example notebook here (https://climetlab.readthedocs.io/en/latest/examples/03-source-cds.html) on my laptop as well as provided binder link.

The code works fine to plot 2m temperature and mean sea level as shown in the example notebook. However, the moment I change one of the requested variables, the notebook code gets stuck indefinitely with the following message. 

source = cml.load_source(
    "cds",
    "reanalysis-era5-single-levels",
    variable=["2t", "blh"],
    product_type="reanalysis",
    area=[50, -50, 20, 50],
    date="2012-12-12",
    time="12:00",
    format="netcdf",
)
for s in source:
    cml.plot_map(s)

2021-10-25 19:31:26,858 INFO Lock 140013250624784 acquired on /tmp/climetlab-jovyan/cdsretriever-d1b901f874b7ac4e804d713920fc5c40fa5f10d2ca221b395896a72f6c9ca41b.nc.lock
2021-10-25 19:31:26,962 INFO Welcome to the CDS
2021-10-25 19:31:26,962 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2021-10-25 19:31:27,080 INFO Request is queued

The message above is from the binder notebook but I get the same result on my laptop as well.

floriankrb commented 3 years ago

I could not reproduce the issue on my machine or in binder. Could you try again?

Note that the priority policy of the CDS (https://cds.climate.copernicus.eu/live/priorities) may make you wait for a long time depending on your previous downloads (or the CDS server has an overload yesterday at this hour).

gewitterblitz commented 3 years ago

I see! Yes, the priority policy would make sense because I was requesting data files multiple times in the same session.

Just curious: What are the recommended practices to access CDS data if one is just doing exploratory analysis involving different datasets in the same session (e.g., single-level, mean, and pressure-levels variables). In other words, when one doesn't know what files would they need beforehand.

floriankrb commented 3 years ago

There are some efficiency tips compiled here. Notice also that some parts of ERA5 can be accessed faster than other

At the end of the day, either you wait until you know which data you need, or you download it ahead of time and you may download useless data (which may not be so much a problem and CliMetLab should cache it nicely). Also, you may want to give a try to the interactive CDS API to choose you data.

gewitterblitz commented 3 years ago

Thanks for sharing the relevant links, definitely helpful. I think the only downside to interactive CDS API (or even climetlab) would be having to store data locally. Python CDS API on the other hand allows lazy loading, am i right?

floriankrb commented 3 years ago

I am not sure I follow you

AFAIK, the CDS API (https://cds.climate.copernicus.eu/api-how-to): import cdsapi c = cdsapi.Client() would still download the data.

While using the toolbox, what I call the interactive CDS API (https://cds.climate.copernicus.eu/toolbox-editor/ , you need to register): import cdstoolbox as ct ... would run only on the CDS machine and would not download data (to your local machine).