search

ajdamico / lodown

locally download and prepare publicly-available microdata
GNU General Public License v3.0
97 stars 47 forks source link

Error in unzip(...) : 'exdir' does not exist #115

Closed rmaranhao closed 7 years ago

rmaranhao commented 7 years ago

Hi.

First, congratulations on what appears to be a very useful package. Unfortunately I cannot even run the examples. I keep getting unzip errors.

I have pasted a full output, believing it to be more helpful. If you want any details I´ll be glad to answer. My goal is to analyze Brazilian PNADC data, but the same error happens with any lodown command I issue.

From the man page, I got the commands: acs_cat <- get_catalog( "acs" , output_dir = file.path( path.expand( "~" ) , "ACS" ) ) lodown( "acs" , acs_cat[ acs_cat$year == 2013 , ] )

The acs_cat data is built, but the lodown command exits with Error in unzip(...) : 'exdir' does not exist

I have already removed varsion 3.3 and installed version 3.4.2 (as of now, the latest) of R

Full output below.

Thanks in advance, Roberto.

R version 3.4.2 (2017-09-28) -- "Short Summer" Copyright (C) 2017 The R Foundation for Statistical Computing Platform: x86_64-w64-mingw32/x64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R.

acs_cat <- get_catalog( "acs" , output_dir = file.path( path.expand( "~" ) , "ACS" ) ) building catalog for acs

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2005

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2006

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2007

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2008

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2009

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2010

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2011

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2012

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2013

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2014

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2015

loading acs catalog from https://www2.census.gov/programs-surveys/acs/data/pums/2016

lodown( "acs" , acs_cat[ acs_cat$year == 2013 , ] ) locally downloading acs

'https://www2.census.gov/programs-surveys/acs/data/pums/2013/1-Year/unix_hwy.zip' cached in 'r://89e2f51b03c949b541d2ec86c83d5f4d.Rcache' copying to 'r:\RtmpQvIxOD\filed304f815761'

R version 3.4.2 (2017-09-28) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale: [1] LC_COLLATE=Portuguese_Brazil.1252 LC_CTYPE=Portuguese_Brazil.1252
[3] LC_MONETARY=Portuguese_Brazil.1252 LC_NUMERIC=C
[5] LC_TIME=Portuguese_Brazil.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] devtools_1.13.3 lodown_0.1.0

loaded via a namespace (and not attached): [1] Rcpp_0.12.13 codetools_0.2-15 XML_3.98-1.9 withr_2.1.0 digest_0.6.12
[6] bitops_1.0-6 R6_2.2.2 DBI_0.7 git2r_0.19.0 magrittr_1.5
[11] httr_1.3.1 stringi_1.1.5 curl_3.0 MonetDBLite_0.4.1 xml2_1.1.1
[16] tools_3.4.2 stringr_1.2.0 selectr_0.3-1 RCurl_1.95-4.8 compiler_3.4.2
[21] rvest_0.3.2 memoise_1.1.0

lodown is now exiting unexpectedly. websites that host publicly-downloadable microdata change often and sometimes those changes cause this software to break. if the error call stack below appears to be a hiccup in your internet connection, then please verify your connectivity and retry the download. otherwise, please open a new issue at https://github.com/ajdamico/asdfree/issues with the contents of this error call stack and also the output of your sessionInfo().

[[1]] lodown("acs", acs_cat[acs_cat$year == 2013, ])

[[2]] withCallingHandlers(catalog <- load_fun(data_name = data_name, catalog, ...), error = function(e) { print(sessionInfo()) if (grepl("cannot allocate vector of size", e)) message(memory_note) else if (grepl("parameter must be specified", e)) message(parameter_note) else if (grepl("to install", e)) message(installation_note) else { message(unknown_error_note) print(sys.calls()) } })

[[3]] load_fun(data_name = data_name, catalog, ...)

[[4]] unzip_warn_fail(tf, exdir = paste0(tempdir(), "/unzips"))

[[5]] tryCatch({ unzip(...) }, warning = function(w) stop(conditionMessage(w)))

[[6]] tryCatchList(expr, classes, parentenv, handlers)

[[7]] tryCatchOne(expr, names, parentenv, handlers[[1L]])

[[8]] doTryCatch(return(expr), name, parentenv, handler)

[[9]] unzip(...)

[[10]] .handleSimpleError(function (e) { print(sessionInfo()) if (grepl("cannot allocate vector of size", e)) message(memory_note) else if (grepl("parameter must be specified", e)) message(parameter_note) else if (grepl("to install", e)) message(installation_note) else { message(unknown_error_note) print(sys.calls()) } }, "'exdir' does not exist", quote(unzip(...)))

[[11]] h(simpleError(msg, call))

Error in unzip(...) : 'exdir' does not exist

ajdamico commented 7 years ago

hi, what happens when you type dir.create(paste0( tempdir(),'/unzips'),showWarnings=TRUE) in the same session?

rmaranhao commented 7 years ago

R shows no errors or warnings. The directory is created.

ajdamico commented 7 years ago

could you run this?

library(lodown)
tf <- tempfile()
cachaca( 'https://www2.census.gov/programs-surveys/acs/data/pums/2013/1-Year/unix_hwy.zip' , tf , mode = 'wb' , filesize_fun = 'httr' )

unzip( tf , exdir = paste0( tempdir() , "/unzips" ) )

lodown:::unzip_warn_fail( tf , exdir = paste0( tempdir() , "/unzips" ) )

warnings()
rmaranhao commented 7 years ago

library(lodown) tf <- tempfile() cachaca( 'https://www2.census.gov/programs-surveys/acs/data/pums/2013/1-Year/unix_hwy.zip' , tf , mode = 'wb' , filesize_fun = 'httr' ) 'https://www2.census.gov/programs-surveys/acs/data/pums/2013/1-Year/unix_hwy.zip' cached in 'r://89e2f51b03c949b541d2ec86c83d5f4d.Rcache' copying to 'r:\Rtmp6Rqpk2\filec0c7f1e7b72'

unzip( tf , exdir = paste0( tempdir() , "/unzips" ) ) Error in unzip(tf, exdir = paste0(tempdir(), "/unzips")) : 'exdir' does not exist

lodown:::unzip_warn_fail( tf , exdir = paste0( tempdir() , "/unzips" ) ) Error in unzip(...) : 'exdir' does not exist

warnings() Mensagem de aviso: In gzfile(file, "rb") : cannot open compressed file 'r:\Rtmp6Rqpk2/pnadc 2016 03.rds', probable reason 'No such file or directory'

A file called filec0c7f1e7b72 (no extension) was created.

rmaranhao commented 7 years ago

In the page: https://stackoverflow.com/questions/15226150/r-exdir-does-not-exist-error

The is this suggestion: dir.create("directory") unzip("file-to-unzip.zip", exdir = "directory/")

It seems to be related to Windows.

ajdamico commented 7 years ago

hi, i probably cannot help you debug this until you figure out why these four lines do not work on your computer, sorry :/

tf <- tempfile()
download.file( 'https://www2.census.gov/programs-surveys/acs/data/pums/2013/1-Year/unix_hwy.zip' , tf , mode = 'wb' )
unzip( tf , exdir = file.path( tempdir() , "unzips" ) )
list.files( file.path( tempdir() , "unzips" ) , full.names = TRUE )

my windows machine unzips them properly..

[1] "C:\\Users\\anthonyd\\AppData\\Local\\Temp\\RtmpOa9GKd/unzips/ACS2013_PUMS_README.pdf"
[2] "C:\\Users\\anthonyd\\AppData\\Local\\Temp\\RtmpOa9GKd/unzips/psam_h56.sas7bdat"
rmaranhao commented 7 years ago

Anthony, I don't know if this will help, but tempdir() seems to return an incorrectly escaped file path.

Example below. I modified your code to output the path passed as exdir. Please note there were four backward slashes on the path created by tempdir, but sometimes the web displays only 2. If I replace them with a forward slash the unzip line runs normally. 


> tf <- tempfile()
> download.file( 'https://www2.census.gov/programs-surveys/acs/data/pums/2013/1-Year/unix_hwy.zip' , tf , mode = 'wb' )
trying URL 'https://www2.census.gov/programs-surveys/acs/data/pums/2013/1-Year/unix_hwy.zip'
Content type 'application/zip' length 630801 bytes (616 KB)
downloaded 616 KB

> path = file.path( tempdir() , "unzips" )
> path
[1] "r:\\\\RtmpKS6m8U/unzips"
> path = file.path('r:/RtmpKS6m8U/unzips')
> path
[1] "r:/RtmpKS6m8U/unzips"
> unzip( tf , exdir =  path)
> list.files( file.path( tempdir() , "unzips" ) , full.names = TRUE )
[1] "r:\\\\RtmpKS6m8U/unzips/ACS2013_PUMS_README.pdf"
[2] "r:\\\\RtmpKS6m8U/unzips/psam_h56.sas7bdat"
ajdamico commented 7 years ago

hi, you'll probably need to ask on stackoverflow? that behavior with the \\\\ doesn't make sense to me..

rmaranhao commented 7 years ago

Anthony, thanks for all the support.

I still don't know why the error occurred, but I found the culprit - ramdrive.

As my boot drive is SSD, I've been using a ramdrive as my temp drive. When I stopped using it (after some hours of debugging) the problem stopped.

I'll use the computer as is for now (everything seems ok, and the performance downgrade is minimal) and worry about SDD degradation later.

Thanks again, Roberto.

Um grande abraço, Roberto.

-- Roberto Maranhão - Maranhão Consultoria r.maranhao@gmail.com (5531) 8482 4456

2017-11-07 15:13 GMT-02:00 Anthony Damico notifications@github.com:

hi, you'll probably need to ask on stackoverflow? that behavior with the \\ doesn't make sense to me..

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ajdamico/lodown/issues/115#issuecomment-342553948, or mute the thread https://github.com/notifications/unsubscribe-auth/ABt1F5zKOrHQf9GKK_1xz6PBoc6ony98ks5s0I-wgaJpZM4QTRhJ .