Errors Preventing Table Download

[jrgant]

I've run into in an error message I can't seem to shake:

Error in if (url.test["statusMessage"] != "OK") { : missing value where TRUE/FALSE needed In addition: Warning message: In (function (endyear, span = 5, dataset = "acs", keyword, table.name, : XML variable lookup tables for this request seem to be missing from ' https://api.census.gov/data/2015/acs5/variables.xml '; temporarily downloading and using archived copies instead; since this is *much* slower, recommend running acs.tables.install()

Trying to run the following query:

geo.ri <- geo.make(state = "RI")

ri.dat <- acs.fetch(geography = geo.ri, endyear = 2015, table.number = "B01001", col.names = "pretty")

Re: the warning message, I've run acs.tables.install() but continue to receive the message regarding the missing XML tables.

Thanks very much. This is a great package.

[eglenn]

Sorry to hear about this trouble. Can you let me know the following?

1) What version of the package are you using, and did you recently upgrade?

2) Can you confirm that you've obtained and installed and API key from the census?

3) Do you know whether your R environment is able to use TLS1.2, which is required by the new API?

If you have access to a command line and can try the following two commands, that might help diagnose this last issue:

curl --tlsv1.2 --verbose "https://api.census.gov/data/2015/acs5?get=NAME,B01003_001E,B01003_001M&for=state:*"

curl --verbose "https://api.census.gov/data/2015/acs5?get=NAME,B01003_001E,B01003_001M&for=state:*"

Thanks.

On Mon, 17 Jul 2017 09:14:14 -0400, Andy Cochran wrote:

[1 <text/plain; UTF-8 (7bit)>] [2 <text/html; UTF-8 (7bit)>] I've run into in an error message I can't seem to shake:

Error in if (url.test["statusMessage"] != "OK") { : missing value where TRUE/FALSE needed In addition: Warning message: In (function (endyear, span = 5, dataset = "acs", keyword, table.name, : XML variable lookup tables for this request seem to be missing from ' https://api.census.gov/data/2015/acs5/variables.xml '; temporarily downloading and using archived copies instead; since this is much slower, recommend running acs.tables.install ()

Trying to run the following query:

`geo.ri <- geo.make(state = "RI")

ri.date <- acs.fetch(geography = geo.ri, endyear = 2015, table.number = "B01001", col.names = "pretty")`

Re: the warning message, I've run acs.tables.install() but continue to receive the message regarding the missing XML tables.

Thanks very much. This is a great package.

― You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

[jrgant]

Thanks very much for the quick response.

1) Version 2.1.0

2) API key has been received and installed. For the ACS package, I think this is my first install.

3) I'm not sure re: the R environment. I'm running R 3.3.3 in RStudio. Do I need to check a curl setting?

4) The commands you sent appear to have gone through. I see the variable information printed to the console after running. I can provide the output if needed.

[eglenn]

On Mon, 17 Jul 2017 09:42:59 -0400, Andy Cochran wrote:

4. The commands you sent appear to have gone through. I see the variable information printed to the console after running. I can provide the output if needed.

Can you send me the output of this, in R:

url.exists("https://api.census.gov/data/2015/acs5?get=NAME,B01003_001E,B01003_001M&for=state:*", .header=T, verbose=T)

Thanks.

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

[jrgant]

Here it is:

* Trying 148.129.75.191... * Connected to api.census.gov (148.129.75.191) port 443 (#0) * successfully set certificate verify locations: * CAfile: C:/Users/Jason/Documents/R/win-library/3.3/RCurl/etc/ca-bundle.crt CApath: none * error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure * Closing connection 0 [1] FALSE

[eglenn]

PS: Also, this:

curlVersion()

On Mon, 17 Jul 2017 10:37:09 -0400, Ezra Haber Glenn wrote:

On Mon, 17 Jul 2017 09:42:59 -0400, Andy Cochran wrote:

4. The commands you sent appear to have gone through. I see the variable information printed to the console after running. I can provide the output if needed.

Can you send me the output of this, in R:

url.exists("https://api.census.gov/data/2015/acs5?get=NAME,B01003_001E,B01003_001M&for=state:*", .header=T, verbose=T)

Thanks.

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

[jrgant]

$age
[1] 3

$version
[1] "7.40.0"

$vesion_num
[1] 468992

$host
[1] "x86_64-pc-win32"

$features
      ssl      libz      ntlm asynchdns    spnego largefile       idn      sspi 
        4         8        16       128       256       512      1024      2048 

$ssl_version
[1] "OpenSSL/1.0.0o"

$ssl_version_num
[1] 0

$libz_version
[1] "1.2.8"

$protocols
 [1] "dict"   "file"   "ftp"    "ftps"   "gopher" "http"   "https"  "imap"   "imaps"  "ldap"   "pop3"   "pop3s"  "rtmp"   "rtsp"   "scp"    "sftp"   "smtp"   "smtps"  "telnet" "tftp"  

$ares
[1] ""

$ares_num
[1] 0

$libidn
[1] ""

[eglenn]

From what I can tell based on this and the experiences of other users, this seems to be the trouble:

$ssl_version [1] "OpenSSL/1.0.0o"

In order to use TLS1.2 (required by the census API with their new https protocol), you need OpenSSL 1.0.1 or higher.

This also explains the output line in your other email that said:

• error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Are you able to update OpenSSL to a newer version?

-_Ezra

On Mon, 17 Jul 2017 12:51:03 -0400, Andy Cochran wrote:

[1 <text/plain; UTF-8 (7bit)>] [2 <text/html; UTF-8 (quoted-printable)>] $age [1] 3

$version [1] "7.40.0"

$vesion_num [1] 468992

$host [1] "x86_64-pc-win32"

$features ssl libz ntlm asynchdns spnego largefile idn sspi 4 8 16 128 256 512 1024 2048

$ssl_version [1] "OpenSSL/1.0.0o"

$ssl_version_num [1] 0

$libz_version [1] "1.2.8"

$protocols [1] "dict" "file" "ftp" "ftps" "gopher" "http" "https" "imap" "imaps" "ldap" "pop3" "pop3s" "rtmp" "rtsp" "scp" "sftp" "smtp" "smtps" "telnet" "tftp"

$ares [1] ""

$ares_num [1] 0

$libidn [1] ""

― You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

[jrgant]

I thought that might be the issue after Googling around a bit. Unfortunately, I'm having a little trouble figuring out how to upgrade.

I'll give that a shot and report back as to whether the issue persists. Thanks for taking the time to troubleshoot.

[eglenn]

OK, great.

(Also, just FYI, it looks like everyone should be updgraing OpenSSL to 1.0.1g or better, due to that "heartbleed" vulnerability that was in the news a while back; see http://heartbleed.com/)

On Mon, 17 Jul 2017 13:37:15 -0400, Andy Cochran wrote:

[1 <text/plain; UTF-8 (7bit)>] [2 <text/html; UTF-8 (quoted-printable)>] I thought that might be the issue after Googling around a bit. Unfortunately, I'm having a little trouble figuring out how to upgrade.

I'll give that a shot and report back as to whether the issue persists. Thanks for taking the time to troubleshoot.

― You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

[eglenn]

Here's one other possibility: although my package doesn't require the "openssl" package, it looks like that package has a system requirement on CRAN that says:

SystemRequirements: OpenSSL >= 1.0.1

So maybe if you install (or upgrade) that package (install.packages("openssl"), it will force some sort of R-accessible installation of the more modern openssl library?

Worth a try?

--Ezra

On Mon, 17 Jul 2017 13:37:15 -0400, Andy Cochran wrote:

[1 <text/plain; UTF-8 (7bit)>] [2 <text/html; UTF-8 (quoted-printable)>] I thought that might be the issue after Googling around a bit. Unfortunately, I'm having a little trouble figuring out how to upgrade.

I'll give that a shot and report back as to whether the issue persists. Thanks for taking the time to troubleshoot.

― You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

[jrgant]

Thanks for pointing to those.

I actually tried reinstalling openssl earlier. While that package's configuration points to an up-to-date version (1.1.0c), as does curl, RCurl continues to report an out-of-date version

[eglenn]

Andy:

Rather than keep beating our heads against the wall here, I'm trying a workaround:

I've made a new version of the acs package using httr instead of RCurl. (The folks at CRAN agree that RCurl is problematic, and recommend a switch like this as well to help with Windows folks.)

Would you be willing to test it out on your setup?

I've uploaded this test version 2.1.1 to http://web.mit.edu/eglenn/www/acs/acs_2.1.1.tar.gz. It's a .gz file, but you should be able to install it without unpacking or unzipping. If you download it first, you can install with:

install.packages(pkgs="acs_2.1.1.tar.gz", clean=T)

If you want to try to do this directly over the internet, I think you can try:

install.packages(pkgs="http://web.mit.edu/eglenn/www/acs/acs_2.1.1.tar.gz", clean=T)

Then just

library(acs)

and test it out.

(Don't forget to run acs.tables.install() as well, and reinstall or api.key.migrate your api key if it doesn't seem to get migrated during install.)

--Ezra

On Mon, 17 Jul 2017 17:19:55 -0400, Andy Cochran wrote:

[1 <text/plain; UTF-8 (7bit)>] [2 <text/html; UTF-8 (7bit)>] Thanks for pointing to those.

I actually tried reinstalling openssl earlier. While that package's configuration points to an up-to-date version (1.1.0c), as does curl, RCurl continues to report an out-of-date version

― You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

[eglenn]

PS: You may need to do this first, since R won't manage all the dependencies on a custom package like this:

install.packages("httr")

On Wed, 19 Jul 2017 13:25:24 -0400, Ezra Haber Glenn wrote:

Andy:

Rather than keep beating our heads against the wall here, I'm trying a workaround:

I've made a new version of the acs package using httr instead of RCurl. (The folks at CRAN agree that RCurl is problematic, and recommend a switch like this as well to help with Windows folks.)

Would you be willing to test it out on your setup?

I've uploaded this test version 2.1.1 to http://web.mit.edu/eglenn/www/acs/acs_2.1.1.tar.gz. It's a .gz file, but you should be able to install it without unpacking or unzipping. If you download it first, you can install with:

install.packages(pkgs="acs_2.1.1.tar.gz", clean=T)

If you want to try to do this directly over the internet, I think you can try:

install.packages(pkgs="http://web.mit.edu/eglenn/www/acs/acs_2.1.1.tar.gz", clean=T)

Then just

library(acs)

and test it out.

(Don't forget to run acs.tables.install() as well, and reinstall or api.key.migrate your api key if it doesn't seem to get migrated during install.)

--Ezra

On Mon, 17 Jul 2017 17:19:55 -0400, Andy Cochran wrote:

[1 <text/plain; UTF-8 (7bit)>] [2 <text/html; UTF-8 (7bit)>] Thanks for pointing to those.

I actually tried reinstalling openssl earlier. While that package's configuration points to an up-to-date version (1.1.0c), as does curl, RCurl continues to report an out-of-date version

― You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

-- Ezra Haber Glenn, AICP Department of Urban Studies and Planning Massachusetts Institute of Technology 77 Massachusetts Ave., Room 7-346 Cambridge, MA 02139 eglenn@mit.edu http://dusp.mit.edu/faculty/ezra-glenn 617.253.2024 (w) 617.721.7131 (c)

[jrgant]

I had to add repo = NULL to the install command after downloading, but that worked! I'm able to grab the ACS table I was looking for.

Thank you for sticking with this one. Very much appreciated.

(FYI, I updated to R 3.4.1 since first writing and am running Windows 10, in case that's relevant at all.)