search

blaylockbk / Herbie

Download numerical weather prediction datasets (HRRR, RAP, GFS, IFS, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.
https://herbie.readthedocs.io/
MIT License
507 stars 75 forks source link

Can't download data on 18th October 2022 between fxx 282 to 360 #346

Closed SaundersJE97 closed 3 months ago

SaundersJE97 commented 4 months ago

I am attempting to download enfo ifs data for the month of October 2022. I'm not able to find data from fxx on the 18th November 282 to 360. Here is the code to repeat the problem I'm encountering.

import numpy as np
from herbie import Herbie
import datetime

date_time = datetime.datetime(year=2022, month=10, day=18)
for forecast_lead_time_fxx in (np.arange(61) * 6):
    forecast_lead_time_fxx = int(forecast_lead_time_fxx)
    H = Herbie(f"{date_time.year}-{date_time.month}-{date_time.day} {hour_str}:00", model="ifs", product="enfo", fxx=forecast_lead_time_fxx)
    try:
        H.index_as_dataframe
    except Exception as e:
        print(f"{date_time.year}-{date_time.month}-{date_time.day} {hour_str}:00", forecast_lead_time_fxx, 'not available')

With the result of: 

 2022-10-18 00:00 282 not available
2022-10-18 00:00 288 not available
2022-10-18 00:00 294 not available
2022-10-18 00:00 300 not available
2022-10-18 00:00 306 not available
2022-10-18 00:00 312 not available
2022-10-18 00:00 318 not available
2022-10-18 00:00 324 not available
2022-10-18 00:00 330 not available
2022-10-18 00:00 336 not available
2022-10-18 00:00 342 not available
2022-10-18 00:00 348 not available
2022-10-18 00:00 354 not available
2022-10-18 00:00 360 not available
SaundersJE97 commented 4 months ago

Not sure if I'm doing something incorrectly or potentially if the data isn't present for those fxx values.

blaylockbk commented 4 months ago

Hi @SaundersJE97

Using rclone , I only see the following files for this date on Azure, and the lead times >= 282 are indeed not available

rclone ls ecmwf-azure:ecmwf/20221018/00z/0p4-beta/enfo
2578180356 20221018000000-0h-enfo-ef.grib2
   983513 20221018000000-0h-enfo-ef.index
2650633506 20221018000000-102h-enfo-ef.grib2
   992307 20221018000000-102h-enfo-ef.index
2650633506 20221018000000-105h-enfo-ef.grib2
   992317 20221018000000-105h-enfo-ef.index
2650633506 20221018000000-108h-enfo-ef.grib2
   992308 20221018000000-108h-enfo-ef.index
2650633506 20221018000000-111h-enfo-ef.grib2
   992318 20221018000000-111h-enfo-ef.index
2650633506 20221018000000-114h-enfo-ef.grib2
   992314 20221018000000-114h-enfo-ef.index
2650633506 20221018000000-117h-enfo-ef.grib2
   992316 20221018000000-117h-enfo-ef.index
2650633506 20221018000000-120h-enfo-ef.grib2
   992313 20221018000000-120h-enfo-ef.index
2650633506 20221018000000-123h-enfo-ef.grib2
   992317 20221018000000-123h-enfo-ef.index
2650633506 20221018000000-126h-enfo-ef.grib2
   992309 20221018000000-126h-enfo-ef.index
2650633506 20221018000000-129h-enfo-ef.grib2
   992318 20221018000000-129h-enfo-ef.index
2650633506 20221018000000-12h-enfo-ef.grib2
   988126 20221018000000-12h-enfo-ef.index
2650633506 20221018000000-132h-enfo-ef.grib2
   992310 20221018000000-132h-enfo-ef.index
2650633506 20221018000000-135h-enfo-ef.grib2
   992318 20221018000000-135h-enfo-ef.index
2650633506 20221018000000-138h-enfo-ef.grib2
   992311 20221018000000-138h-enfo-ef.index
2650633506 20221018000000-141h-enfo-ef.grib2
   992321 20221018000000-141h-enfo-ef.index
2650633506 20221018000000-144h-enfo-ef.grib2
   992308 20221018000000-144h-enfo-ef.index
2650633506 20221018000000-150h-enfo-ef.grib2
   992310 20221018000000-150h-enfo-ef.index
2650633506 20221018000000-156h-enfo-ef.grib2
   992314 20221018000000-156h-enfo-ef.index
2650633506 20221018000000-15h-enfo-ef.grib2
   988166 20221018000000-15h-enfo-ef.index
2650633506 20221018000000-162h-enfo-ef.grib2
   992307 20221018000000-162h-enfo-ef.index
2650633506 20221018000000-168h-enfo-ef.grib2
   992308 20221018000000-168h-enfo-ef.index
2650633506 20221018000000-174h-enfo-ef.grib2
   992307 20221018000000-174h-enfo-ef.index
2650633506 20221018000000-180h-enfo-ef.grib2
   992309 20221018000000-180h-enfo-ef.index
2650633506 20221018000000-186h-enfo-ef.grib2
   992309 20221018000000-186h-enfo-ef.index
2650633506 20221018000000-18h-enfo-ef.grib2
   988127 20221018000000-18h-enfo-ef.index
2650633506 20221018000000-192h-enfo-ef.grib2
   992305 20221018000000-192h-enfo-ef.index
2650633506 20221018000000-198h-enfo-ef.grib2
   992308 20221018000000-198h-enfo-ef.index
2650633506 20221018000000-204h-enfo-ef.grib2
   992305 20221018000000-204h-enfo-ef.index
2650633506 20221018000000-210h-enfo-ef.grib2
   992305 20221018000000-210h-enfo-ef.index
2650633506 20221018000000-216h-enfo-ef.grib2
   992309 20221018000000-216h-enfo-ef.index
2650633506 20221018000000-21h-enfo-ef.grib2
   988149 20221018000000-21h-enfo-ef.index
2650633506 20221018000000-222h-enfo-ef.grib2
   992305 20221018000000-222h-enfo-ef.index
2650633506 20221018000000-228h-enfo-ef.grib2
   992306 20221018000000-228h-enfo-ef.index
2650633506 20221018000000-234h-enfo-ef.grib2
   992307 20221018000000-234h-enfo-ef.index
2650633506 20221018000000-240h-enfo-ef.grib2
   992310 20221018000000-240h-enfo-ef.index
2650633506 20221018000000-246h-enfo-ef.grib2
   992360 20221018000000-246h-enfo-ef.index
2650633506 20221018000000-24h-enfo-ef.grib2
   988127 20221018000000-24h-enfo-ef.index
2650633506 20221018000000-252h-enfo-ef.grib2
   992365 20221018000000-252h-enfo-ef.index
2650633506 20221018000000-258h-enfo-ef.grib2
   992359 20221018000000-258h-enfo-ef.index
2650633506 20221018000000-264h-enfo-ef.grib2
   992369 20221018000000-264h-enfo-ef.index
2650633506 20221018000000-270h-enfo-ef.grib2
   992362 20221018000000-270h-enfo-ef.index
2650633506 20221018000000-276h-enfo-ef.grib2
   992356 20221018000000-276h-enfo-ef.index
2650633506 20221018000000-27h-enfo-ef.grib2
   988156 20221018000000-27h-enfo-ef.index
2650633506 20221018000000-30h-enfo-ef.grib2
   988128 20221018000000-30h-enfo-ef.index
2650633506 20221018000000-33h-enfo-ef.grib2
   988158 20221018000000-33h-enfo-ef.index
2650633506 20221018000000-36h-enfo-ef.grib2
   988130 20221018000000-36h-enfo-ef.index
2650633506 20221018000000-39h-enfo-ef.grib2
   988152 20221018000000-39h-enfo-ef.index
2650633506 20221018000000-3h-enfo-ef.grib2
   983982 20221018000000-3h-enfo-ef.index
2650633506 20221018000000-42h-enfo-ef.grib2
   988131 20221018000000-42h-enfo-ef.index
2650633506 20221018000000-45h-enfo-ef.grib2
   988149 20221018000000-45h-enfo-ef.index
2650633506 20221018000000-48h-enfo-ef.grib2
   988130 20221018000000-48h-enfo-ef.index
2650633506 20221018000000-51h-enfo-ef.grib2
   988148 20221018000000-51h-enfo-ef.index
2650633506 20221018000000-54h-enfo-ef.grib2
   988131 20221018000000-54h-enfo-ef.index
2650633506 20221018000000-57h-enfo-ef.grib2
   988150 20221018000000-57h-enfo-ef.index
2650633506 20221018000000-60h-enfo-ef.grib2
   988128 20221018000000-60h-enfo-ef.index
2650633506 20221018000000-63h-enfo-ef.grib2
   988159 20221018000000-63h-enfo-ef.index
2650633506 20221018000000-66h-enfo-ef.grib2
   988128 20221018000000-66h-enfo-ef.index
2650633506 20221018000000-69h-enfo-ef.grib2
   988152 20221018000000-69h-enfo-ef.index
2650633506 20221018000000-6h-enfo-ef.grib2
   983942 20221018000000-6h-enfo-ef.index
2650633506 20221018000000-72h-enfo-ef.grib2
   988129 20221018000000-72h-enfo-ef.index
2650633506 20221018000000-75h-enfo-ef.grib2
   988136 20221018000000-75h-enfo-ef.index
2650633506 20221018000000-78h-enfo-ef.grib2
   988126 20221018000000-78h-enfo-ef.index
2650633506 20221018000000-81h-enfo-ef.grib2
   988133 20221018000000-81h-enfo-ef.index
2650633506 20221018000000-84h-enfo-ef.grib2
   988124 20221018000000-84h-enfo-ef.index
2650633506 20221018000000-87h-enfo-ef.grib2
   988137 20221018000000-87h-enfo-ef.index
2650633506 20221018000000-90h-enfo-ef.grib2
   988125 20221018000000-90h-enfo-ef.index
2650633506 20221018000000-93h-enfo-ef.grib2
   988141 20221018000000-93h-enfo-ef.index
2650633506 20221018000000-96h-enfo-ef.grib2
   988125 20221018000000-96h-enfo-ef.index
2650633506 20221018000000-99h-enfo-ef.grib2
   988136 20221018000000-99h-enfo-ef.index
2650633506 20221018000000-9h-enfo-ef.grib2
   983984 20221018000000-9h-enfo-ef.index
      108 LICENCE.txt
      198 README.txt

Hope that answers your question.

Pro tip:

Herbie accepts datetime objects for the date, it doesn't have to be a string (i just implemented that for my own convenience)

date_time = datetime.datetime(year=2022, month=10, day=18)
for forecast_lead_time_fxx in np.arange(61) * 6:
    forecast_lead_time_fxx = int(forecast_lead_time_fxx)
    H = Herbie(
        date_time,
        model="ifs",
        product="enfo",
        fxx=forecast_lead_time_fxx,
    )
blaylockbk commented 3 months ago

I'm going to close this for now, but please reopen if you learn anything new or see other problems.