search

ajdamico / asdfree

analyze survey data for free
http://asdfree.com/
GNU General Public License v3.0
612 stars 449 forks source link

fix censo_escolar and restore 2009 #294

Closed ajdamico closed 6 years ago

ajdamico commented 7 years ago

i think it should turn green on sisyphus once you debug? thanks cc @hannesmuehleisen

hannes commented 7 years ago

Don't think it has anything to do with bugs, it fails importing ftp://ftp.inep.gov.br/microdados/micro_censo_escolar2009.zip on both platforms.

ajdamico commented 7 years ago

i've commented out the 2009 download until we have a chance to fix..

https://github.com/ajdamico/lodown/commit/6a9deb6703669250ac66e2793e3326d16e85920f

2017-10-17 15:31:12 censo_escolar catalog entry 14 of 23 stored in '/export/scratch1/home/hannes/eanthony/r-wd/censo_escolar-1508238377/MonetDB'
2017-10-17 15:31:12 
2017-10-17 15:47:56 'ftp://ftp.inep.gov.br/microdados/micro_censo_escolar2009.zip'
2017-10-17 15:47:56 cached in
2017-10-17 15:47:56 '/export/scratch1/home/hannes/eanthony/r-tmp/censo_escolar/2e62704af3aa2a7ad545429c20b4f2fb.Rcache'
2017-10-17 15:47:56 copying to
2017-10-17 15:47:56 '/export/scratch1/home/hannes/eanthony/r-tmp/censo_escolar/Rtmp2hbfpv/file24af24bd5707'
2017-10-17 15:47:56 
2017-10-17 15:48:20 QQ: 'SELECT COUNT(*) FROM matricula_2009'
2017-10-17 15:48:20 
2017-10-17 15:48:20 
2017-10-17 15:48:20 lodown is now exiting unexpectedly.
2017-10-17 15:48:20 websites that host publicly-downloadable microdata change often and sometimes those changes cause this software to break.
2017-10-17 15:48:20 if the error call stack below appears to be a hiccup in your internet connection, then please verify your connectivity and retry the download.
2017-10-17 15:48:20 otherwise, please open a new issue at `https://github.com/ajdamico/asdfree/issues` with the contents of this error call stack and also the output of your `sessionInfo()`.
2017-10-17 15:48:20 
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[1]]
2017-10-17 15:48:20 source(lodown::syntaxtractor("censo_escolar", replacements = machine_specific_replacements, 
2017-10-17 15:48:20     setup_test = "setup"), echo = TRUE)
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[2]]
2017-10-17 15:48:20 withVisible(eval(ei, envir))
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[3]]
2017-10-17 15:48:20 eval(ei, envir)
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[4]]
2017-10-17 15:48:20 eval(ei, envir)
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[5]]
2017-10-17 15:48:20 lodown("censo_escolar", output_dir = file.path("/export/scratch1/home/hannes/eanthony/r-wd/censo_escolar-1508238377"))
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[6]]
2017-10-17 15:48:20 withCallingHandlers(catalog <- load_fun(data_name = data_name, 
2017-10-17 15:48:20     catalog, ...), error = function(e) {
2017-10-17 15:48:20     if (grepl("cannot allocate vector of size", e)) 
2017-10-17 15:48:20         message(memory_note)
2017-10-17 15:48:20     else if (grepl("parameter must be specified", e)) 
2017-10-17 15:48:20         message(parameter_note)
2017-10-17 15:48:20     else if (grepl("to install", e)) 
2017-10-17 15:48:20         message(installation_note)
2017-10-17 15:48:20     else {
2017-10-17 15:48:20         message(unknown_error_note)
2017-10-17 15:48:20         print(sys.calls())
2017-10-17 15:48:20     }
2017-10-17 15:48:20 })
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[7]]
2017-10-17 15:48:20 load_fun(data_name = data_name, catalog, ...)
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[8]]
2017-10-17 15:48:20 DBI::dbGetQuery(db, paste0("SELECT COUNT(*) FROM matricula_", 
2017-10-17 15:48:20     catalog[i, "year"]))
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[9]]
2017-10-17 15:48:20 DBI::dbGetQuery(db, paste0("SELECT COUNT(*) FROM matricula_", 
2017-10-17 15:48:20     catalog[i, "year"]))
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[10]]
2017-10-17 15:48:20 .local(conn, statement, ...)
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[11]]
2017-10-17 15:48:20 dbSendQuery(conn, statement, ...)
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[12]]
2017-10-17 15:48:20 dbSendQuery(conn, statement, ...)
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[13]]
2017-10-17 15:48:20 .local(conn, statement, ...)
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[14]]
2017-10-17 15:48:20 stop("Unable to execute statement '", statement, "'.\nServer says '", 
2017-10-17 15:48:20     env$message, "'.")
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[15]]
2017-10-17 15:48:20 .handleSimpleError(function (e) 
2017-10-17 15:48:20 {
2017-10-17 15:48:20     if (grepl("cannot allocate vector of size", e)) 
2017-10-17 15:48:20         message(memory_note)
2017-10-17 15:48:20     else if (grepl("parameter must be specified", e)) 
2017-10-17 15:48:20         message(parameter_note)
2017-10-17 15:48:20     else if (grepl("to install", e)) 
2017-10-17 15:48:20         message(installation_note)
2017-10-17 15:48:20     else {
2017-10-17 15:48:20         message(unknown_error_note)
2017-10-17 15:48:20         print(sys.calls())
2017-10-17 15:48:20     }
2017-10-17 15:48:20 }, "Unable to execute statement 'SELECT COUNT(*) FROM matricula_2009'.\nServer says 'ParseException:SQLparser:42S02!SELECT: no such table 'matricula_2009''.", 
2017-10-17 15:48:20     quote(.local(conn, statement, ...)))
2017-10-17 15:48:20 
2017-10-17 15:48:20 [[16]]
2017-10-17 15:48:20 h(simpleError(msg, call))
2017-10-17 15:48:20 
2017-10-17 15:48:20 Error in .local(conn, statement, ...) : 
2017-10-17 15:48:20   Unable to execute statement 'SELECT COUNT(*) FROM matricula_2009'.
2017-10-17 15:48:20 Server says 'ParseException:SQLparser:42S02!SELECT: no such table 'matricula_2009''.
2017-10-17 15:48:20 Calls: source ... <Anonymous> -> .local -> dbSendQuery -> dbSendQuery -> .local
2017-10-17 15:48:20 In addition: There were 50 or more warnings (use warnings() to see the first 50)
2017-10-17 15:48:20 Execution halted
ajdamico commented 7 years ago

this will break because of duplicates

> devtools::install_github("ajdamico/lodown")
Downloading GitHub repo ajdamico/lodown@master
from URL https://api.github.com/repos/ajdamico/lodown/zipball/master
Installing lodown
"C:/PROGRA~1/R/R-34~1.2/bin/x64/R" --no-site-file --no-environ --no-save  \

  "C:/Users/anthonyd/AppData/Local/Temp/Rtmp2bLsva/devtools1648727338e8/ajdamico-lodown-2172b1b"  \
  --library="C:/Users/anthonyd/Documents/R/win-library/3.4" --install-tests 

* installing *source* package 'lodown' ...
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
*** arch - i386
*** arch - x64
* DONE (lodown)
> lodown::get_catalog("censo_escolar",output_dir=tempdir())
building catalog for censo_escolar

   year                                                      dbfolder                                              output_folder                                                                            full_url
1  1995 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/1995                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar1995.zip
2  1996 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/1996                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar1996.zip
3  1997 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/1997                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar1997.zip
4  1998 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/1998                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar1998.zip
5  1999 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/1999                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar1999.zip
6  2000 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2000                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar2000.zip
7  2001 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2001                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar2001.zip
8  2002 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2002                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar2002.zip
9  2003 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2003                        ftp://ftp.inep.gov.br/microdados/micro_censo_escolar2003.zip
27 2003 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2003  ftp://ftp.inep.gov.br/microdados/Arquivos_antigos/microdados_censo_escolar2003.zip
21 2004 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2004                  ftp://ftp.inep.gov.br/microdados/microdados_censo_escolar_2004.zip
24 2004 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2004       ftp://ftp.inep.gov.br/microdados/Arquivos_antigos/micro_censo_escolar2004.zip
22 2005 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2005                  ftp://ftp.inep.gov.br/microdados/microdados_censo_escolar_2005.zip
23 2006 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2006                ftp://ftp.inep.gov.br/microdados/microdados_censo_escolar_2006_2.zip
25 2006 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2006       ftp://ftp.inep.gov.br/microdados/Arquivos_antigos/micro_censo_escolar2006.zip
11 2007 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2007                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2007.zip
12 2008 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2008                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2008.zip
28 2008 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2008 ftp://ftp.inep.gov.br/microdados/Arquivos_antigos/microdados_censo_escolar_2008.zip
13 2009 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2009                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2009.zip
26 2009 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2009      ftp://ftp.inep.gov.br/microdados/Arquivos_antigos/micro_censo_escolar_2009.zip
14 2010 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2010                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2010.zip
15 2011 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2011                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2011.zip
16 2012 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2012                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2012.zip
17 2013 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2013                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2013.zip
18 2014 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2014                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2014.zip
19 2015 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2015                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2015.zip
20 2016 C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/MonetDB C:\\Users\\anthonyd\\AppData\\Local\\Temp\\Rtmp2bLsva/2016                       ftp://ftp.inep.gov.br/microdados/micro_censo_escolar_2016.zip
guilhermejacob commented 7 years ago

rewriting it

guilhermejacob commented 7 years ago

reopening for now. waiting for test