wesm / feather

Feather: fast, interoperable binary data frame storage for Python, R, and more powered by Apache Arrow
Apache License 2.0
2.74k stars 167 forks source link

Large data frames can be saved but not loaded #372

Closed shntnu closed 5 years ago

shntnu commented 5 years ago

Large data frames can be saved but not loaded (feather v0.3.2)

dummy <- as.data.frame(matrix(rnorm(30000000, 30000, 1000)))
feather::write_feather(dummy, "dummy.feather")
dummy <- feather::read_feather("dummy.feather")
#> Error: evaluation nested too deeply: infinite recursion / options(expressions=)?

Created on 2019-02-06 by the reprex package (v0.2.1)

jameslamb commented 5 years ago

@shntnu it's possible you are seeing the same error as in #374 . Unfortunately feather has a dependency on tibble and you need certain combinations of tibble and feather together to get it working.

Could you try upgrading to tibble > 2.0.0 and see if you're able to reproduce this issue?

I did not have any problems running your code with this environment (from sessionInfo()):

R version 3.5.0 (2018-04-23)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS  10.14

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

loaded via a namespace (and not attached):
 [1] compiler_3.5.0  hms_0.4.2       tools_3.5.0     pillar_1.3.1   
 [5] rstudioapi_0.7  tibble_2.0.1    yaml_2.2.0      crayon_1.3.4   
 [9] Rcpp_1.0.0.1    feather_0.3.2   pkgconfig_2.0.2 rlang_0.3.1

@hadley why does feather need the tibble dependency? I think it might make more sense to have read_feather() read up into base data frames and then to add a from_feather() or something like that to the tibble package. Curious to hear your thoughts.

shntnu commented 5 years ago

@jameslamb tibble 2.1.1 + feather 0.3.3 worked! Thanks for diagnosing.