qread error with large file

ccshao commented 5 years ago

I have a 4Gb R object and saved it to disk with qsave (3.7 Gb in disk), but later could not read it with qread. Playing with the qread parameter does not solve the problem. I am using the latest version of qs.

qs::qsave(x, "x.q")
xx1 <- qs::qread("x.q")
Error in c_qread(normalizePath(file, mustWork = FALSE), use_alt_rep, strict,  : 
  read error

Session info: R version 3.6.1 (2019-07-05) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Linux Mint 19.1

Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=de_DE.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=de_DE.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] qs_0.17.3 Seurat_3.0.2 dplyr_0.8.3 colorout_1.2-1

loaded via a namespace (and not attached): [1] httr_1.4.0 tidyr_0.8.3 jsonlite_1.6
[4] viridisLite_0.3.0 splines_3.6.1 lsei_1.2-0
[7] R.utils_2.9.0 gtools_3.8.1 Rdpack_0.11-0
[10] assertthat_0.2.1 ggrepel_0.8.1 globals_0.12.4
[13] pillar_1.4.2 lattice_0.20-38 reticulate_1.13
[16] glue_1.3.1 digest_0.6.20 RColorBrewer_1.1-2 [19] SDMTools_1.1-221.1 colorspace_1.4-1 cowplot_1.0.0
[22] htmltools_0.3.6 Matrix_1.2-17 R.oo_1.22.0
[25] plyr_1.8.4 pkgconfig_2.0.2 bibtex_0.4.2
[28] tsne_0.1-3 listenv_0.7.0 purrr_0.3.2
[31] scales_1.0.0 RANN_2.6.1 gdata_2.18.0
[34] Rtsne_0.15 RApiSerialize_0.1.0 tibble_2.1.3
[37] ggplot2_3.2.0 withr_2.1.2 ROCR_1.0-7
[40] pbapply_1.4-1 lazyeval_0.2.2 survival_2.44-1.1
[43] magrittr_1.5 crayon_1.3.4 R.methodsS3_1.7.1
[46] future_1.14.0 nlme_3.1-140 MASS_7.3-51.4
[49] gplots_3.0.1.1 ica_1.0-2 tools_3.6.1
[52] fitdistrplus_1.0-14 data.table_1.12.2 gbRd_0.4-11
[55] stringr_1.4.0 plotly_4.9.0 munsell_0.5.0
[58] cluster_2.1.0 irlba_2.3.3 compiler_3.6.1
[61] rsvd_1.0.2 caTools_1.17.1.2 rlang_0.4.0
[64] grid_3.6.1 ggridges_0.5.1 htmlwidgets_1.3
[67] igraph_1.2.4.1 labeling_0.3 bitops_1.0-6
[70] npsurv_0.4-0 gtable_0.3.0 codetools_0.2-16
[73] littleutils_0.1.0 reshape2_1.4.3 R6_2.4.0
[76] gridExtra_2.3 zoo_1.8-6 future.apply_1.3.0 [79] KernSmooth_2.23-15 metap_1.1 ape_5.3
[82] stringi_1.4.3 parallel_3.6.1 Rcpp_1.0.2
[85] sctransform_0.2.0 png_0.1-7 tidyselect_0.2.5
[88] lmtest_0.9-37

traversc commented 5 years ago

Hi, would it be possible to show me an example I can reproduce or share your data?

ccshao commented 5 years ago

Is that ok I put the whole qs saved file in the dropbox and provide you the link?

traversc commented 5 years ago

Yes, thanks. You can e-mail me the link if you don't want to make the data completely public.

traversc commented 5 years ago

The error was determined for very large S4 objects (> 2 GB). Will be fixed in the next version.

ccshao commented 5 years ago

Thanks for the quick fix!

qsbase / qs

qread error with large file #14