GEFS URL for 1.0 to 0.5 Degree Output Transition

[lesserwhirls]

Greetings,

It's totally awesome that rnoaa is pulling data from the Unidata THREDDS data server. One thing to note is that the GEFS dataset will be changing from 1.0 degree output to 0.5 degree output in the near future (TBD, but slated for 2015 - http://www.nco.ncep.noaa.gov/pmb/changes/).

I'll be happy to submit a pull request with the new url once the transition is happened.

Cheers!

[sckott]

Thanks for the heads up. Please do submit a PR when it happens. Thanks :+1:

[sckott]

@lesserwhirls looks like the functions for working with THREDDS are broken now. Has the change been made?

cc @potterzot

[sckott]

an example request

http://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GEFS/Global_1p0deg_Ensemble/members/GEFS_Global_1p0deg_Ensemble_20151129_0000.grib2

gives

Error {
    code = 400;
    message = "Unrecognized request";
}

[lesserwhirls]

Hi @sckott - the example url you have listed above is not a valid OPeNDAP request. If, for example, you request the dods dataset description:

http://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GEFS/Global_1p0deg_Ensemble/members/GEFS_Global_1p0deg_Ensemble_20151129_0000.grib2.dds

the request works. The transition to GEFS 0.5 degree has not happened yet, so the 1.0 degree should still be working.

Is there a another example request which isn't working?

[sckott]

@lesserwhirls Okay, thanks. Hmm, I'll have a look, maybe I constructed the URL wrong.

[lesserwhirls]

It would be good to get the full url being requested. I do not know R at all, otherwise I'd go digging around to see.

On Mon, Nov 30, 2015 at 4:16 PM, Scott Chamberlain <notifications@github.com

wrote:

@lesserwhirls https://github.com/lesserwhirls Okay, thanks. Hmm, I'll have a look, maybe I constructed the URL wrong.

— Reply to this email directly or view it on GitHub https://github.com/ropensci/rnoaa/issues/109#issuecomment-160793348.

[potterzot]

@sckott is this broken in the GEFS R functions currently? I can look into it this evening to see what's wrong if so.

[sckott]

@potterzot yeah, all tests fail, and all egs don't work - if they truly are broken, we should get a new patch version up to CRAN soon. I'll open a new issue so we don't conflate with the topic of this one

[potterzot]

Just submitted a PR that should address #119 and #117.

With regards to the resolution change, maybe it would be worth adding the ability for the user to select the resolution, since it seems both the 1 and 0.5 degree GEFS data will be available.

@lesserwhirls if you provide the new URL for the 0.5, I'll add it to gefs.R so that the user can select the resolution.

@sckott, figured it would be better to submit two separate PRs, since the 0.5 resolution may not drop for a few more days.

Thanks!

[sckott]

Sounds good

[lesserwhirls]

Yikes, this means the 0.5 degree gefs is being sent to us with the 1 degree gefs data header. How long has the test been failing? Hopefully not too long...

There will be a new url for the 0.5 degree gefs, likely later today. I'll let you know when it's live. The United for the 1.0 degree gefs will continue to work until the old data ages off the system (about 2 weeks).

Sean

On Monday, November 30, 2015, Scott Chamberlain notifications@github.com wrote:

Sounds good

— Reply to this email directly or view it on GitHub https://github.com/ropensci/rnoaa/issues/109#issuecomment-160866443.

[lesserwhirls]

Scratch that, this is probably why I need coffee before getting on guthub. I'll continue monitoring the data feeds to see when the 0.5 gefs becomes available.

On Tuesday, December 1, 2015, Sean Arms sarms@ucar.edu wrote:

Yikes, this means the 0.5 degree gefs is being sent to us with the 1 degree gefs data header. How long has the test been failing? Hopefully not too long...

There will be a new url for the 0.5 degree gefs, likely later today. I'll let you know when it's live. The United for the 1.0 degree gefs will continue to work until the old data ages off the system (about 2 weeks).

Sean

On Monday, November 30, 2015, Scott Chamberlain <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Sounds good

— Reply to this email directly or view it on GitHub https://github.com/ropensci/rnoaa/issues/109#issuecomment-160866443.

[sckott]

thanks @lesserwhirls

[sckott]

There will be a new url for the 0.5 degree gefs, likely later today. I'll let you know when it's live.

@lesserwhirls @potterzot will this happen soon? Hoping to push a new version of this pkg to CRAN. I can hold off if it's soon, but otherwise I'll push today probably, then we can submit a new version to CRAN soon with the changes

[lesserwhirls]

I dug a bit deeper, and we won't be getting the 0.5 degree gefs yet - it's not being transmitted via noaaport, so we wil look at other options (which will take more time...could be months).

Sorry about that. The 1.0 degree should be good to go, though.

On Tuesday, December 1, 2015, Scott Chamberlain notifications@github.com wrote:

There will be a new url for the 0.5 degree gefs, likely later today. I'll let you know when it's live.

@lesserwhirls https://github.com/lesserwhirls @potterzot https://github.com/potterzot will this happen soon? Hoping to push a new version of this pkg to CRAN. I can hold off if it's soon, but otherwise I'll push today probably, then we can submit a new version to CRAN soon with the changes

— Reply to this email directly or view it on GitHub https://github.com/ropensci/rnoaa/issues/109#issuecomment-161142559.

[sckott]

ok, thanks @lesserwhirls

[potterzot]

It could happen tonight if the URL is ready. Otherwise tomorrow. I can make the changes tonight and then drop in the new link when it becomes available. On Dec 1, 2015 4:41 PM, "Scott Chamberlain" notifications@github.com wrote:

There will be a new url for the 0.5 degree gefs, likely later today. I'll let you know when it's live.

@lesserwhirls https://github.com/lesserwhirls @potterzot https://github.com/potterzot will this happen soon? Hoping to push a new version of this pkg to CRAN. I can hold off if it's soon, but otherwise I'll push today probably, then we can submit a new version to CRAN soon with the changes

— Reply to this email directly or view it on GitHub https://github.com/ropensci/rnoaa/issues/109#issuecomment-161142559.

[sckott]

@potterzot but @lesserwhirls just said it could be months, i'm not sure i follow, maybe we're talking about 2 diff. things?

[potterzot]

Ah, sorry, I missed that msg. Good to go then! On Dec 1, 2015 5:10 PM, "Scott Chamberlain" notifications@github.com wrote:

@potterzot https://github.com/potterzot but @lesserwhirls https://github.com/lesserwhirls just said it could be months, i'm not sure i follow, maybe we're talking about 2 diff. things?

— Reply to this email directly or view it on GitHub https://github.com/ropensci/rnoaa/issues/109#issuecomment-161147237.

[sckott]

submitting soon

[sckott]

@lesserwhirls @potterzot any update on this?

[potterzot]

@sckott @lesserwhirls AFAIK it isn't available yet, so we can't incorporate it. At least on the unidata thredds server, they only provide the 1 degree option.

[sckott]

okay, thanks!

[lesserwhirls]

Sorry for the silence on this end. NCEP has implemented the resolution change, but it's not yet coming through their data feed to us (called CONDUIT). We are actually hearing an overview of model changes at this very moment, so I may have some news soon.

On Mon, Apr 11, 2016 at 11:10 AM, Nicholas Potter notifications@github.com wrote:

@sckott https://github.com/sckott @lesserwhirls https://github.com/lesserwhirls AFAIK it isn't available yet, so we can't incorporate it. At least on the unidata thredds server http://thredds.ucar.edu/thredds/idd/forecastModels.html, they only provide the 1 degree option.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/ropensci/rnoaa/issues/109#issuecomment-208452981

[sckott]

great, thx for the update @lesserwhirls

[sckott]

hey @lesserwhirls @potterzot - these functions are breaking again. Seems the thredds urls are 404ing. Having URLs changed?

[lesserwhirls]

Hey @sckott - we are having major issues with the data array on the machine currently hosing thredds.ucar.edu. We've put in a DNS request to change to a backup machine, so as soon as that goes through we should be rockin' and rollin' again. Thanks for the report!

[potterzot]

@sckott, do you think it would be worth adding a check/error message so that if a user tries to pull some data and gets an inability to connect error they would know to try again later?

Also, I have some similar functionality now for CFS and NMME data. These are longer-term climate forecast datasets that are also accessible via netcdf. If you think it would be worth adding I'd be happy to put together a pull request.

[sckott]

do you think it would be worth adding a check/error message so that if a user tries to pull some data and gets an inability to connect error they would know to try again later?

yeah, makes sense. I don't think ncdf4 has the flexibility to error well, right? could use httr to ping the url, and see if you get a 200

Also, I have some similar functionality now for CFS and NMME data. These are longer-term climate forecast datasets that are also accessible via netcdf. If you think it would be worth adding I'd be happy to put together a pull request.

Yeah, sounds great

[sckott]

closing as gefs functions are now defunct