A problem/ Using ECMWF wind data

[amirheidari]

Dear all, I want to use ECMWF wind data sets in PyGnome. When I execute the code, I get a Value Error: timestamp out of range for platform localtime()/gmtime() function. For your information, I have used Pygnome with GFS wind data set, and it was OK, but the ECMWF wind data sets lead to an Value Error.

I have attached my script and the input files. I will appreciate if anyone can help me.

Gulf - Copy.zip

[ChrisBarker-NOAA]

Hmm,

I just took a look at the files. It looks like they are converted from grib, and have the time specified as:

    time:units = "hours since 1900-01-01 00:00:0.0" ;
    time:long_name = "time" ;
    time:calendar = "gregorian" ;

in the C++ code, we are using the platform localtime()/gmtime() functions -- and, python uses them for datetime.datetime.

Apparently, they can't deal with dates from 1900.

in fact, when I try your scrit on OS-X, I get:

""" The Windows function localtime() does not accept dates earlier than January 1, 1970 GMT """ and then, when it tries to run:

RuntimeError: No available data in the time interval that is being modeled Model time: 2016-01-15 00:00:00 Mover: GridWindMover of type <class 'gnome.movers.wind_movers.GridWindMover'> Data available from 2085-12-30 16:00:00 to 2086-01-30 16:00:00

So it's clearly getting the wrong dates! (they should be around 2016 some time)

What platform are you on? It may be that this could actually work on modern 64 bit platforms.

@coconnor8 -- can you take a look?

In any case, if you transform the netcdf file to use a reference date after 1970 -- (say the year 2000), then it should work.

-CHB

[jay-hennen]

No, this is not going to be compatible at all. First issue is that it's 'rectangular grid' format which isn't supported yet (lat and lon are 1D)

Secondly, the time series is not ascending in time. It goes from Jan 1-31, then back to Jan 1, and then ends on Feb 1

On the upside, the dates seemed to be parsed correctly from the file, so the fact that the dates are hours from 1900 does not seem to be an issue.

[ChrisBarker-NOAA]

@amirheidari: it looks like time-shifting the time variable is the way to go for now.

@jay-hennen: let's put it on the queue to support rectangular grids in gridded... Here's our first us case.

[coconnor8]

I already responded to this issue on the regular GNOME mailing list.

[jay-hennen]

Not only time shifted, it needs to also be ascending only.

[amirheidari]

Dear All, Thank you very much for you comments. @ChrisBarker-NOAA: I am using Windows 8 (64bit). The data time epoch was modified (as attached). @coconnor8 : I wrote a code to change the time epoch. The time data (attached: new_ecmwf.nc) are now measured as hours from 2015. @jay-hennen: In the new data set, the time data are increasing and I have one new data every 6 hours.

I have attached the old data set, the conversion code and the new data set. Now I get an OS error: "GridWindMover_c.TextRead returned an error".

I think this new error is more fundamental than the previous one, which was value error.

Is there a problem with the format of the new file (NetCDF4)? The old data set was NetCDF-3!

The new data set, with all the corrections, is wind data for Jan. 2016 (if you want to execute the model with new data set, please change the start time of the spill in the script).

Hope to hear from you Sincerely, Payam

NewNC.zip

[amirheidari]

Dear all, Thank you very much for your contribution. Using NCO I converted the NetCDF4 to NetCDF3. The model ran successfully!

For new researchers: NCO has a Windows .exe version. The NCO code in command line:

cd C:\nco ## goes to directory where NCO is installed ncks -3 old_file.nc new_file.nc

Everybody, thank you again for your support. Now, every researcher can use ECMWF wind data in PyGnome. I think ECMWF is the only provider of global long-term wind data sets in .nc format. When downloading from ECMWF, pay attention that if you check wrong boxes in the data request form (some are related to analysis data and others related to forecast data), the time array will not be in order.

If you have comments about the reliability of using ECMWF wind data sets (with rectangular grid) in PyGnome, please let me know.

Regards Payam Amir-Heidari

[ChrisBarker-NOAA]

Thanks for posting your solution.

where did you get your ECWMF dataset?

[amirheidari]

Dear Dr. Barker, The ECMWF website is: www.ecmwf.int/ It has different models. We use ERA-Interim data set: http://apps.ecmwf.int/datasets/data/interim-full-daily/levtype=sfc/

Different data are available from 1979 to now. u10 and v10 are the wind velocity components at height of 10 meters. The wind data, especially the new data (after 2006), is very reliable. The model documentation can be found in the website. The following link is also useful for choosing your data time step: https://software.ecmwf.int/wiki/pages/viewpage.action?pageId=56658233 For downloading data, the user should register in the website. The user can select the region of interest, and the grid size. The smallest grid size is 0.125 degrees, which is really useful for modern modelings.

Sincerely, Payam