Open Yefee opened 10 years ago
rsignell-usgs
commented
10 years ago I might be missing something, but it seems to me you have answered your own question: The 1st dataset has 24 hour forecasts The 2nd dataset has 48 hour forecasts The 3rd dataset has both 24 and 48 hour forecasts, so is twice as big.
-Rich
Yefee
commented
10 years ago yeah, im just wondering if it is a bug. Please test and fix it, thank you！
来自 MEIZU MX
-------- 原始邮件 -------- 发件人：Rich Signell notifications@github.com 时间：周六 1月18日 01:20 收件人：nctoolbox/nctoolbox nctoolbox@noreply.github.com 抄送：Yefee hec1990@qq.com 主题：Re: [nctoolbox] Using the nctoolbox results in the missing of TIGGE DATA？ (#27)
I might be missing something, but it seems to me you have answered your own question: The 1st dataset has 24 hour forecasts The 2nd dataset has 48 hour forecasts The 3rd dataset has both 24 and 48 hour forecasts, so is twice as big.
-Rich
Reply to this email directly or view it on GitHub: https://github.com/nctoolbox/nctoolbox/issues/27#issuecomment-32625279
rsignell-usgs
commented
10 years ago I'm sorry, what is the bug? The 3rd file has twice as many grib records as the other 2. You can check independently with programs like "wgrib2".
-Rich
On Fri, Jan 17, 2014 at 12:30 PM, Yefee notifications@github.com wrote:
yeah, im just wondering if it is a bug. Please test and fix it, thank you！
来自 MEIZU MX
-------- 原始邮件 -------- 发件人：Rich Signell notifications@github.com 时间：周六 1月18日 01:20 收件人：nctoolbox/nctoolbox nctoolbox@noreply.github.com 抄送：Yefee hec1990@qq.com 主题：Re: [nctoolbox] Using the nctoolbox results in the missing of TIGGE DATA？ (#27)
I might be missing something, but it seems to me you have answered your own question: The 1st dataset has 24 hour forecasts The 2nd dataset has 48 hour forecasts The 3rd dataset has both 24 and 48 hour forecasts, so is twice as big.
-Rich
Reply to this email directly or view it on GitHub: https://github.com/nctoolbox/nctoolbox/issues/27#issuecomment-32625279
— Reply to this email directly or view it on GitHubhttps://github.com/nctoolbox/nctoolbox/issues/27#issuecomment-32626475 .
Dr. Richard P. Signell (508) 457-2229 USGS, 384 Woods Hole Rd. Woods Hole, MA 02543-1598
rsignell-usgs
commented
10 years ago Wait, I see what you mean. Even though there are twice as many time records in the 3rd dataset, NCTOOLBOX thinks there is just one more.
This might be a problem interpreting the ensemble dimension.
I'm pretty sure we haven't encountered data with that yet.
-Rich
hohonuuli
commented
10 years ago Sorry, I don't know enough about grib to be able to say if it's a bug with NetCDF or an error with the grib file itself. Can you verify that the bad grib works as expected in some other tool?
I tried using the grib_api tools to look into the grib structure but it's a loooooooong dump of text that means nothing to me.
rsignell-usgs
commented
10 years ago This seems to be a problem with netCDF-Java. If you open Tools-UI in Java, and then open your GRIB datasets as FeatureTypes=>Grids you can see the same thing:
Thanks, Rich
hohonuuli
commented
10 years ago Yes, but do we have confirmation that the bug isn't with the grib file? It needs to be opened with some grib specific tool to verify that the format is actually what @Yefee expects. It's conceivable that there is something wrong with the grib file itself, file size alone isn't enough to validate it. Can someone extract some data from those missing grib records? Until that's confirmed, we can't assume the bug is in NetCDF. (Although grib's do seem to be a huge source of headaches ...)
rsignell-usgs
commented
10 years ago Uh, I did look at them with WGRIB2, a GRIB specific tool.
hohonuuli
commented
10 years ago Got it. I updated my comment. Can you extract some data from those missing records with a grib tool? Just to make doubly sure.
rsignell-usgs
commented
10 years ago I think I see what's going on here.
NetCDF-Java (and NCTOOLBOX) are reading the data just fine, but the grib records are getting slotted into the prediction time values. So a 48 hour forecasts from day 0 gets assigned the same timestamp as a 24 hour forecast from day 1. And that's why the file with both 24 and 48 hour forecasts only has 12 entries instead of 11. All the 48 hour forecasts just overwrite the 24 hour forecasts except for the last one, so you get one more time record.
rsignell-usgs
commented
10 years ago @Yefee , if you could split the problematic file into two grib files containing the 24hour and 48hour forecast grib files, and then read them using NCTOOLBOX, I think you would be okay.
If you used wgrib2 it would look like this:
wgrib2 -match ":48 hour" mars-web239-20140117150636-1554-0681.grib -grib 48hour.grib
wgrib2 -match ":24 hour" mars-web239-20140117150636-1554-0681.grib -grib 24hour.grib@cwardgar, does my explanation of what's going on sound correct? If so, it might be useful to know about this issue in case some other NetCDF-Java user encounters it.
cwardgar
commented
10 years ago Hi Rich,
I haven't worked with GRIB yet, so I'm not the one to ask about this. John might want to know though.
On Fri, Jan 17, 2014 at 2:39 PM, Rich Signell notifications@github.comwrote:
@cwardgar https://github.com/cwardgar, does my explanation of what's going on sound correct? If so, it might be useful to know about this issue in case some other NetCDF-Java user encounters it.
— Reply to this email directly or view it on GitHubhttps://github.com/nctoolbox/nctoolbox/issues/27#issuecomment-32651061 .
rsignell-usgs
commented
10 years ago @JohnLCaron, does my explanation of what's going on sound correct? If so, it might be useful to know about this issue in case some other NetCDF-Java user encounters it.
JohnLCaron
commented
10 years ago what version of netcdf-java is the toolbox using?
rsignell-usgs
commented
10 years ago I tried opening these grib files in Tools-UI, using NetCDF-Java 4.3.20.
hohonuuli
commented
10 years ago @JohnLCaron ... and the NetCDF version in the toolbox is currently 4.3.15.
JohnLCaron
commented
10 years ago hohonuuli
commented
10 years ago rsignell-usgs
commented
10 years ago Okay, I tossed those grib files in here: http://geoport.whoi.edu/thredds/catalog/usgs/data2/rsignell/data/catalog.html
I also put in the grib files 24hour.grib and 48hour.grib, which I split out of mars-web239-20140117150636-1554-0681.grib
On Fri, Jan 17, 2014 at 6:09 PM, John Caron notifications@github.comwrote:
getting a 404 on these:
http://download.ecmwf.int/data/237/mars-web237-20140117144653-6370-0756.grib
http://download.ecmwf.int/data/237/mars-web237-20140117144903-6371-0784.grib
http://download.ecmwf.int/data/239/mars-web239-20140117150636-1554-0681.grib
can you make them available?
— Reply to this email directly or view it on GitHubhttps://github.com/nctoolbox/nctoolbox/issues/27#issuecomment-32661503 .
Dr. Richard P. Signell (508) 457-2229 USGS, 384 Woods Hole Rd. Woods Hole, MA 02543-1598
JohnLCaron
commented
10 years ago ok, yes, your interpretation is correct. 4.3 just creates 1D time based on forecasst time. version 4.5 will give you a 2D time dataset( ref/valid). however thats not yet released.
http://download.ecmwf.int/data/237/mars-web237-20140117144653-6370-0756.grib http://download.ecmwf.int/data/237/mars-web237-20140117144903-6371-0784.grib http://download.ecmwf.int/data/239/mars-web239-20140117150636-1554-0681.grib
Above three data sets are perturb ensemble forecastings from project TIGGE. The first two sets time step are 24 and 48 respectively, and the third one contains both 24 and 48. In addition, the third file size is twice of the former's. All of them have variables: mean sea level pressure, time, lon and so on,meanwhile pressure has 4 dimensions: ensemble.id time lon and lat. However, when I use the toolbox to patch the pressure, I 'm amazed at that the first two file's pressure is pressure(14,11,111,180) and the third is pressure(14,12,111,180); 14 is members of ensemble, 11 and 12 is size of time. So, the question is since the third file size is twice of the former's, why the third one just contains one more section time of pressure? Has something been missing? Apologize for my poor English.