tidyverse / multidplyr

A dplyr backend that partitions a data frame over multiple processes
https://multidplyr.tidyverse.org
Other
641 stars 75 forks source link

qs::qsave error in partition() #82

Closed MarioClueless closed 5 years ago

MarioClueless commented 5 years ago

Hello,

I'm working on a 32 core cluster using:

R version 3.4.3 (2017-11-30)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux

Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so

tidyverse_1.2.1 multidplyr_0.0.0.9000

I am trying to integrate a multidplyr approach to a dplyr pipe since my dataframe has around 45 million groups and I need to fit a model for each of them.

After installing multidplyr (still not very easy with R 3.4) I try to run either a reduced example of my flow or the tutorial from github in my machine.

Nevertheless I am always, and whatever I try fall on this error:

Error in qs::qsave(values, path, preset = "fast", check_hash = FALSE,  : 
  unused arguments (preset = "fast", check_hash = FALSE, nthreads = 2)

For the moment I don't know where it comes from and info is somewhat limited on Google.

Any ideas ?

Thanks in advance !

hadley commented 5 years ago

It works fine on travis, https://travis-ci.org/tidyverse/multidplyr. Given the error, I wonder if you have a bad version version of qs installed? What does the following code return for you?

packageVersion("qs")
qs::qsave
MarioClueless commented 5 years ago

The output is:

> packageVersion("qs")
[1] ‘0.13.1’
> qs::qsave
function (x, file, compress_level = -1L) 
{
    invisible(.Call(`_qs_qsave`, x, file, compress_level))
}
<environment: namespace:qs>
hadley commented 5 years ago

Yeah, that's very old - CRAN qs is at 0.18.3.

MarioClueless commented 5 years ago

Thanks, I guess my system admin has some updating to do.