search

Matt-Brigida / EIAdata

R Wrapper for the Energy Information Administration (EIA) API
16 stars 15 forks source link

Unable to connect without user agent #17

Closed MoreCoffee12 closed 2 years ago

MoreCoffee12 commented 2 years ago

The package has been working great, but after upgrading R I now receive an error when I try to access the data. Here is the example:

library(EIAdata) require(httr) str.symbol.raw = "PET.A103600001.M" key <- "7a2672468131c5a95dffc487d741edf6" ETA103600001M <- getEIA(key=key, ID=str.symbol.raw )

This code returns:

failed to load HTTP resource Error: 1: failed to load HTTP resource

Traceback:

1. getEIA(key = key, ID = str.symbol.raw) 2. .getMonEIA(ID, key = key) 3. xmlParse(file = url, isURL = TRUE) 4. (function (msg, ...) . { . if (length(grep("\\\n$", msg)) == 0) . paste(msg, "\n", sep = "") . if (immediate) . cat(msg) . if (length(msg) == 0) { . e = simpleError(paste(1:length(messages), messages, sep = ": ", . collapse = "")) . class(e) = c(class, class(e)) . stop(e) . } . messages <<- c(messages, msg) . })(character(0))

The URL opens in my web browser so I think my key is valid. Looking at Stack Exchange it seems that this can be a problem if no User Agent is assigned. The case suggest the following work around where the page is accessed with a User Agent.

UA <- "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36" my.url <- "https://api.eia.gov/series/?series_id=PET.A103600001.M&api_key=7a2672468131c5a95dffc487d741edf6&out=xml" page <- GET(my.url, user_agent(UA)) doc <- XML::xmlParse(content(page, "text")) df <- data.frame( date = sapply(doc["//data/row/date"], XML::xmlValue), value = sapply(doc["//data/row/value"], XML::xmlValue) )

And I have data in the df structure. Any idea why I'm seeing the Error: 1: failed to load HTTP resource message?

Matt-Brigida commented 2 years ago

Thanks for opening the issue. I just saw the following notice on the EIA API website. Looks like we'll need to make the switch to https.

Notice: As of February 11st, 2022 The API now only responds to HTTPS. Insecure (HTTP) web protocols are no longer supported.

MoreCoffee12 commented 2 years ago

Thanks for the note back on this, sounds good.

Matt-Brigida commented 2 years ago

I pushed a potential fix to a branch called https_fix. You can install and test the fixed version of EIAdata with:

remove.packages(EIAdata)
library(devtools)
install_github("Matt-Brigida/EIAdata", ref="https_fix")

Loading the package and running the following returns:

getEIA(ID="PET.A103600001.M", key = key)
           PET.A103600001.M
## 1983-01-01          46218.3
## 1983-02-01          48906.1
## 1983-03-01          54111.7
## 1983-04-01          49093.7
## 1983-05-01          52021.3
## 1983-06-01          52517.2

So looks good. If you get a chance to test it with series of different frequencies let me know how it goes. I have not fixed getCatEIA yet, maybe tomorrow.

And the solution (in this commit) doesn't require setting a user agent, but it does use some code you provided when opening your issue. So @MoreCoffee12 thank you for your helpful message! :+1: :beer:

MoreCoffee12 commented 2 years ago

You all are awesome, I just confirmed the fix works! Thank you so much for the quick response on this. The package is very helpful, gets a lot of use from me.

Matt-Brigida commented 2 years ago

That is great to hear! The https_fix branch has been merged to master, and set as release v0.1.3. The v0.1.3 fix is also on its way to CRAN---should be approved and available on CRAN within a couple days.