read.socrata(url, token) neglects my token

[alxfed]

I can not throttle-up my downloads by using the token issued to my app (on data.chicago.com portal, where I registered it)

Error 1:

token <- "___my_app_token__";
fdf <- read.socrata("https://data.cityofchicago.org/resource/7edu-s3u7.csv?$where=station_name=\"Foster Weather Station\"", token)

which returns:

2016-10-06 10:39:53.685 getResponse: 
Error in httr GET: 403  https://data.cityofchicago.org/resource/7edu-s3u7.csv?%24where=station_name%3D%22Foster%20Weather%20Station%22&app_token=%2524%2524app_token%3D___my_app_token_______

I have NO IDEA where did the first 'token' (2524 2524) come from, do you?

Non-error:

fdf <- read.socrata("https://data.cityofchicago.org/resource/7edu-s3u7.csv?$where=station_name=\"Foster Weather Station\"")

WITHOUT A TOKEN (and not throttled-up) works perfectly well!

[geneorama]

Alex,

In your example, why are there two underscores in the protocol section of the URL?

fdf <- read.socrata("h___s://data.cityofchicago.org/resource/7edu-s3u7.csv? $where=station_name=\"Foster Weather Station\"", token)

Also, it's helpful if you format the code as code in your question. I hope you don't mind, but I have taken the liberty of modifying the formatting of your question.

For inline code, you can enclose your code in backticks (the key to the left of the 1 on most keyboards). This will render the surrounded text as code, e.g. print("hello world").

For blocks of code, you can start with a "code fence" which is three backticks. Optionally, you can also indicate your language, e.g. ```R

You are using the > symbol currently, which renders as an inline response. You probably did the same thing I did the first time used github, and copied and pasted directly out of the R Console window.

[alxfed]

Yes, thank you. I routinely change https or http, so that the URL wouldn't get converted to a link; changed it back.

Once again, the problem is: if I don't add any SoQL parameters to the endpoint URL - everything works fine WITH a token and without a token, but if I do add filtering with $where - the version WITH a token (which is not throttled down) stops working, while the version without a token - still works.

Note: and it doesn't matter how you pass the parameter to the function, as ..., app_token="__my_token_") or just as ..., token) if you've assigned it. I tried both of course. :)

[chrismetcalf]

FYI, this is also cross-posted by @alxfed in Stack Overflow: https://stackoverflow.com/questions/39901044/rsocrata-package-with-chicago-data-neglects-my-token/39903084?noredirect=1#comment67096239_39903084

Thanks for helping out @geneorama! 🍻

[tomschenkjr]

I've added a test to check for this in the future.

The test both conducts a where query and uses an app token. It fails, as expected.

read.socrata("https://soda.demo.socrata.com/resource/4334-bgaj.csv?$where=magnitude > 3.0", app_token="ew2rEMuESuzWPqMkyPfOSGJgE")

I'm now reworking the code to make sure this error doesn't happen again.

[alxfed]

Thank you, @tomschenkjr. This particular dataset ( 7edu-s3u7 ) with three weather stations each adding an observation every hour as a row is particularly hard to deal with without where.

[tomschenkjr]

@alxfed - a fix is on the branch issue105 which you can install using install_github() from the devtools package. It's alpha, but seems to work without breaking anything else.

This will be rolled into our release on CRAN, likely available first week of Nov.

[alxfed]

Thank you. Shall do it. I found out that this weather data has unforgivable gaps (like the one from May 9 till May 19 (!!!) this year). Started using a different source (NOAA ASOS ISH/ISD). Thank you for your help anyway, @tomschenkjr, I'm sure I will use query parameters in other projects. Regards.

[tomschenkjr]

@alxfed - Today we've moved the new code (with this and other bug fixes) to the master branch, so I'd recommend grabbing it from there.

The weather data may be inconsistent and unavailable outside of the May 30th-September 30th range (Chicago's beach season). /cc @levyj

[alxfed]

It's all in place except for the gap (and it's for three stations too, somebody was on vacation for 10 days there). Thank you once again, I will take it where you say, @tomschenkjr Regards.