expersso / OECD

Reproducible and programmatic access to OECD data
126 stars 20 forks source link

Problem with proxy config calling get_datasets #11

Open dbradnum opened 6 years ago

dbradnum commented 6 years ago

I'm trying to get this useful package up and running behind a corporate firewall, where we need to specify proxy server details - specifically the proxy server, and to use Windows authentication. I've tackled a few similar issues in the past, and have solutions for packages using RCurlor httr. However, I've been having trouble with the get_datasets() function (which looks like it uses curl, behind read_xml).

I can't see a way to pass through or pre-configure the proxy settings we need as it stands, but I might be able to see a workaround. Any suggestions or advice would be welcome!

To give specific examples:

  1. This code gives a timeout as expected, since no proxy server specified
library(OECD)
dataset_list = get_datasets()
  1. I can specify the server as follows... but then I get an HTTP 407 authentication error
myProxyServer = "..."
Sys.setenv(HTTPS_PROXY = myProxyServer)
dataset_list = get_datasets()

This is also expected, since in our setting we need to specify proxyuserpwd = ":" to work with Windows authentication.

  1. Having looked at the code behind get_datasets, I've figured out that the following adjustment works successfully:
    
    library(httr)
    library(xml2)

set_config(config( proxy = myProxyServer, proxyuserpwd = ":" ))

url = "https://stats.oecd.org/RestSDMX/sdmx.ashx/GetKeyFamily/all" datasets = read_xml(GET(url))



This way, I can use `httr::set_config` to configure the full range of curl options for httr, and then pass the result of the GET into read_xml. The rest of the code behind `get_datasets` works as before.

Can you suggest another way of making this function work with our proxy config, or would you consider making a change so the package uses an httr-based approach? For the moment, I'll create a little helper function with the code above - but obviously I'd prefer not to maintain this separately from the package.

Thanks!
expersso commented 6 years ago

I think https://github.com/expersso/OECD/commit/728ed21abd786d2147e6aedd974e494b3bd968e3 should have fixed it. Would you mind giving it a try?

dbradnum commented 6 years ago

Many thanks for making the change - pleased to say this works well with our proxy config. Sorry for the delay in responding; I needed to test using my work machine, and things have been a bit hectic there!

Are you likely to push this new version up to CRAN any time soon, do you think? We can use it either way, but would be helpful to know what the likely plan is.