tidyverse / multidplyr

A dplyr backend that partitions a data frame over multiple processes
https://multidplyr.tidyverse.org
Other
641 stars 75 forks source link

filter() not observing cluster_assign_value() objects #17

Closed danpbowen closed 8 years ago

danpbowen commented 8 years ago

As I understand how to send objects/values to each cluster for filtering - for example - is to use the cluster_assign_value() function for each value needed.

Maybe I'm not doing this correctly, but I would expect the below code to work.

> cluster2 <- create_cluster(2)
> mtcars_cl <- partition(mtcars, cluster = cluster2)
> cyl8 <- 8
> cluster_assign_vlaue(cluster = cluster2, name = "cyl8", value = a)
> cluster_get(cluster = cluster2, "cyl8")
[[1]]
[1] 8

[[2]]
[1] 8
> mtcars_cl %>% filter(cyl == cly8)
Error in checkForRemoteErrors(lapply(cl, recvResult)) : 
  2 nodes produced errors; first error: object 'cly8' not found
hadley commented 8 years ago

Typo?

danpbowen commented 8 years ago

whoa. I had a couple typos in there. This checks out:

cluster2 <- create_cluster(2)
mtcars_cl <- partition(mtcars, cluster = cluster2)
cyl8 <- 8
cluster_assign_value(cluster = cluster2, name = "cyl8", value = cyl8)
cluster_get(cluster = cluster2, "cyl8")
mtcars_cl %>% filter(cyl == cyl8)