Error in fromJSON

[fozy81]

I'm getting this error:

Error in fromJSON(content, handler, default.size, depth, allowComments, : invalid JSON input

Any ideas? Is the NBN server down?

My code: library(devtools) install_github("rnbn", username = 'JNCC-UK') library(rnbn)

occ <- getOccurrences(tvks = "NBNSYS0000002010")

On Linux LTS 12.04 R - 3.0.3 (also same error on Windows 7, 3.01)

[sckott]

hey @fozy81 I just tried this and it worked for me.

Can you try again now to see if it happens again? And what version of RJSONIO do you have installed? packageVersion("RJSONIO")? I have 1.0.3

[fozy81]

Hi (I've got a NBN gateway name and password and have successfully logged into the NBN gateway site)

Here's the output from my console - same error:

packageVersion("RJSONIO") [1] ‘1.0.3’

################## And I tried the code again:

library(devtools) install_github("rnbn", username = 'JNCC-UK') library(rnbn)

occ <- getOccurrences(tvks = "NBNSYS0000002010")

install_github("rnbn", username = 'JNCC-UK') Installing github repo rnbn/master from JNCC-UK Downloading master.zip from https://github.com/JNCC-UK/rnbn/archive/master.zip Installing package from /tmp/RtmpsMz1W7/master.zip arguments 'minimized' and 'invisible' are for Windows only Installing rnbn '/usr/lib/R/bin/R' --vanilla CMD INSTALL '/tmp/RtmpsMz1W7/devtools7d359cc509f/rnbn-master' \ --library='/home/tim/R/i686-pc-linux-gnu-library/3.0' --install-tests

• installing source package 'rnbn' ... * R * inst * preparing package for lazy loading * help ** installing help indices * building package indices * installing vignettes * testing if installed package can be loaded
• DONE (rnbn) Reloading installed rnbn

  library(rnbn)

  occ <- getOccurrences(tvks = "NBNSYS0000002010") Requesting batch 1 of 1 Error in fromJSON(content, handler, default.size, depth, allowComments, : invalid JSON input

[fozy81]

Thanks for looking into this, looks like a really useful package.

Tim

[sckott]

I'm not sure what's going on. @AugustT Any thoughts?

[fozy81]

Thanks again for reviewing this. Just to confirm, the pop up for entering my username and password did appear and I entered my NBN gateway details.

[AugustT]

Hi Guys,

Tim, is the first time you have used the package? I think this might be a user specific problem as both Scott and I can get this to work on our machines. Can you first try:

NBNCookies(remove=TRUE)

This will remove your cached cookies, please let me know what prints to screen

Then try your request again:

occ <- getOccurrences(tvks = "NBNSYS0000002010")

Let me know what that does.

There is the occasional problem with these cookies, I recently found a problem when they expire but thought I had got a fix in place. I'd like to keep this cookies handling functionality but if it is unreliable I'm tempted to remove it.

Tom

[AugustT]

A note for other people who have problem with cookie handling is that an alternative method to the prompted dialog boxes is to run this line before getOccurrences:

nbnLogin(username = 'myUsername', password = 'myPassword')

[fozy81]

Here's the screen prints:

NBNCookies(remove=TRUE) No cookies file found NULL

occ <- getOccurrences(tvks = "NBNSYS0000002010") Requesting batch 1 of 1 Error in fromJSON(content, handler, default.size, depth, allowComments, : invalid JSON input

Doesn't seem to matter if I put in correct or incorrect password always goes to: Error in fromJSON(content, handler, default.size, depth, allowComments, : invalid JSON input

[AugustT]

Thanks Tim. What do you get if you login with nbnLogin?

nbnLogin(username = 'myUsername', password = 'myPassword')

I have not tested this package on Linux, and wonder if there may be an issue there.

[AugustT]

For information the "Error in fromJSON(content, handler, default.size, depth, allowComments, : invalid JSON input" Error message is indication that R did not get the result from the server that it was expecting, we just need to work out why.

[fozy81]

Hi, same response with nbnLogin(username = 'Tim Foster', password = '*****'):

I think the space in my username maybe the problem?

Error in fromJSON(content, handler, default.size, depth, allowComments, : invalid JSON input

[AugustT]

Hi Tim,

That is a very interesting point and it could well be a problem since this is used in a URL where a space should be noted as '%20'. It may be that I can fix this simply by catching spaces in the username or password (I assume it would also be a problem there) and replace them with %20. I will take this away and see this could explain the error. I will let you know how I get on.

Tom

[fozy81]

just created a new account with username without space

occ <- getOccurrences(tvks = "NBNSYS0000002010") Requesting batch 1 of 1 [1] "Login successful" Error in runnbnurl(service = "obs", tvks = temp_tvks, datasets = datasets, : When interpreting response from NBN servers: invalid JSON input . This can occur if the NBN servers are experiencing problems. If the problem persists report this error to the package maintainers

[AugustT]

Hi Tim,

Well that does suggest that the original problem was indeed because of the space and I will look to implement a fix for that.

The new error is typical of when the servers go down, I usually check on the servers by going here: https://data.nbn.org.uk/ and seeing if the page loads. On this occassion the page looks fine and so I assume the servers are up and running.

When I run the code I get:

occ <- getOccurrences(tvks = "NBNSYS0000002010") Requesting batch 1 of 1 Requesting data providers information IMPORTANT: https://data.nbn.org.uk/Terms By using this package you are agreeing to the Gateway Terms & Conditions and Privacy Policy. This message can be suppressed using the acceptTandC argument

Perhaps try again.

If this still fails I would be interested to see what response you are getting from the server. If you go to the webpage I linked above and then login. Once logged in put this in your browser's address bar and see what you get back:

https://data.nbn.org.uk/api/taxonObservations?ptvk=NBNSYS0000002010

After a little time you should get a list of elements like this one:

{ observationID: 265440796, fullVersion: false, datasetKey: "GA000477", surveyKey: "158", sampleKey: "158-SAMPLE", observationKey: "276705", featureID: 158089, location: "SW72", resolution: "10km", taxonVersionKey: "NBNSYS0000002010", pTaxonVersionKey: "NBNSYS0000002010", pTaxonName: "Isoetes histrix", pTaxonAuthority: "Bory", startDate: "1998-02-01", endDate: "1998-02-28", sensitive: false, absence: false, publicAttribute: false, attrStr: "BRC_CONCEPT¬Vas_1044¦CARDTYPE¬4¦NATIVE_STATUS¬NATIVE¦VICECOUNTY¬West Cornwall (with Scilly)", dateTypekey: "O " },

[fozy81]

Hi Tom,

I'm still getting the 'Error in runnbnurl(service = "obs", tvks = temp_tvks, datasets = datasets, : When interpreting response from NBN servers: invalid JSON input . This can occur if the NBN servers are experiencing problems. If the problem persists report this error to the package maintainers' in R

I loaded and logged in to https://data.nbn.org.uk/

Entered: https://data.nbn.org.uk/api/taxonObservations?ptvk=NBNSYS0000002010

And got a list of features as above.

[AugustT]

Hi Tim,

Thank you for this. I wonder then if this is a linux issue. @sckott are you aware of any differences in the way fromJSON or getURL operate on linux. Could it be that the standard response on linux is .xml not .JSON for example?

I'll have a hunt around for a solution and a linux machine to test on.

Tom

[fozy81]

Sorry, should have mentioned. I'm on my works computer today - WIndows 7 machine (but will try this again when I get back on linux tonight). WIndows 7 machine: RJSONIO = 1.0.3 $platform [1] "x86_64-w64-mingw32" $arch [1] "x86_64" $os [1] "mingw32" $system [1] "x86_64, mingw32" $version.string [1] "R version 3.0.1 (2013-05-16)" $nickname [1] "Good Sport"

[sckott]

@AugustT I do have a linux machine to try on, but I've never figured out how to get TCL/TK installed on linux, which I think also requires X11

One way around this would be to not have the login dialog box, but perhaps there is a good reason for that.

[AugustT]

Hi Tim,

Well this is confusing. The function that throws the error is runnbnurl and is mainly an internal function. It is exported though, so you can run your query like this:

output <- runnbnurl(service="obs", tvks="NBNSYS0000002010")

This should still give you your error.

The object that is causing us the problem is created by this line:

resp <- try(getURL(url, curl = curl), silent=TRUE)

I think you will need to set up a Curl profile like this (assuming you are in the same working directory that you are running your code in):

options(RCurlOptions = list(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl")))
agent = "rnbn v0.1"
options(RCurlOptions = list(sslversion=3L, ssl.verifypeer = FALSE))
curl = getCurlHandle()
cookies <- 'rnbn_cookies.txt'
curlSetOpt(cookiefile = cookies, cookiejar = cookies,
           useragent = agent, followlocation = TRUE, curl=curl)

With this set up you can then run the lines:

url<-'https://data.nbn.org.uk/api/taxonObservations?ptvk=NBNSYS0000002010' resp <- getURL(url, curl = curl)

Then we want to see what this resp object is and why fromJSON(resp, asText=TRUE) doesn't work on it.

[fozy81]

Here's the print out - thanks for looking at this:

output <- runnbnurl(service="obs", tvks="NBNSYS0000002010") [1] "Login successful"

options(RCurlOptions = list(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"))) agent = "rnbn v0.1" options(RCurlOptions = list(sslversion=3L, ssl.verifypeer = FALSE)) curl = getCurlHandle() cookies <- 'rnbn_cookies.txt' curlSetOpt(cookiefile = cookies, cookiejar = cookies,

• useragent = agent, followlocation = TRUE, curl=curl) An object of class "CURLHandle" Slot "ref": <pointer: 0x000000001e135b90>

url<-'https://data.nbn.org.uk/api/taxonObservations?ptvk=NBNSYS0000002010' resp <- getURL(url, curl = curl) resp [1] "[{\"observationID\":265440796,\"fullVersion\":false,\"datasetKey\":\"GA000477\",\"surveyKey\":\"158\",\"sampleKey\":\"158-SAMPLE\",\"observationKey\":\"276705\",\"featureID\":158089,\"location\":\"SW72\",\"resolution\":\"10km\",\"taxonVersionKey\":\"NBNSYS0000002010\",\"pTaxonVersionKey\":\"NBNSYS0000002010\",\"pTaxonName\":\"Isoete..............

[fozy81]

setup the Curl profile appears to fix the problem. Thank you,

Tim

[AugustT]

So the fromJSON and getOccurrences now works?

[AugustT]

@fozy81 can you confirm a few things. Once you ran the curl profile script above did getOccurrences then start working?

If you start a fresh R session does it still work?

If yes to both then my guess would be that the error you got to start with that was due to your username somehow messed up your Curl profile. In which case a simple turn it off and turn it on again might have worked.

[fozy81]

Looks like I just need to set my working directory and then it works. But server is down now I think. Anyway, looks like it was a username issue.

library(rnbn) Loading required package: RCurl Loading required package: bitops Loading required package: RJSONIO Loading required package: tcltk

occ <- getOccurrences(tvks = "NBNSYS0000002010") Requesting batch 1 of 1 [1] "Login successful" Error in runnbnurl(service = "obs", tvks = temp_tvks, datasets = datasets, : When interpreting response from NBN servers: invalid JSON input . This can occur if the NBN servers are experiencing problems. If the problem persists report this error to the package maintainers setwd(U:\R) Error: unexpected input in "setwd(U:\" setwd("U:\R") occ <- getOccurrences(tvks = "NBNSYS0000002010") Requesting batch 1 of 1 Requesting data providers information IMPORTANT: https://data.nbn.org.uk/Terms By using this package yo

[AugustT]

Thanks for the details. One last thing, what was your working directory before you changed it?

[fozy81]

My R working dir on my work machine resets when I restart: "C:/Windows/system32" - i think this is a default but my work machine is virtual/server so doesn't have a C: drive - This isn't an issue on my home machine.