hafen
commented
8 years ago To add to the mystery, talking to @jrounds, we thought maybe it had to do with the rbind(unlist(..., recursive = TRUE)). But check out this example:
# write key: i, value: i for i = 1, ..., 10000
rhwrite(lapply(1:10000, function(x) list(x, x)), file = "/tmp/rhtest")
# map that collects "1" as key and a single-row data-frame
map <- expression({
for(i in seq_along(map.values)) {
rhcollect("1", data.frame(key = map.keys[[i]], val = map.values[[i]], stringsAsFactors = FALSE))
}
})
reduce <- expression(
pre = {
adata <- list()
},
reduce = {
adata[[length(adata) + 1]] <- reduce.values
},
post = {
rhcollect(reduce.key, adata)
}
)
b <- rhwatch(map = map, reduce = reduce, input = "/tmp/rhtest", output = "/tmp/rhtestout")
b <- rhread("/tmp/rhtestout")
head(b[[1]][[2]][[1]])
# [[1]]
# key val
# 1 4000 4000
# [[2]]
# key val
# 1 3999 3999
# [[3]]
# key val
# 1 3998 3998
head(b[[1]][[2]][[2]])
# [[1]]
# key val
# 1 4000 4000
# [[2]]
# key val
# 1 3999 3999
# [[3]]
# key val
# 1 3998 3998
length(b[[1]][[2]][[1]])
# 6000
length(b[[1]][[2]][[2]])
# 4000In this case, it looks like the first pass through reduce$reduce, adata[[1]] is set by reduce.values which has length 6000. But in the second pass, reduce.values has length 4000 and it looks like both adata[[1]][1:4000] and adata[[2]] are set to this reduce.values.
To check this, see:
digest::digest(b[[1]][[2]][[1]])
# "32b71ff2deb98db18ecb67b7b2935dd7"
digest::digest(b[[1]][[2]][[2]])
# "e1597289afb05fdd6a47a3493f20b50e"
digest::digest(b[[1]][[2]][[1]][1:4000])
# "e1597289afb05fdd6a47a3493f20b50e"This verifies that adata[[1]][1:4000] is the same as adata[[2]]. Perhaps this has to do with R only making copies when it thinks it absolutely needs to, such that the assignment of adata[[1]] is a pointer to reduce.values instead of a copy and it is updated in the second pass. Perhaps there's some logic that has changed in the R internals that needs to be reflected in RHIPE's C code. Or maybe I'm way off :).
saptarshiguha
@saptarshiguha, could you take a look at this and see if you can replicate?
The following does not work correctly:
However, if you change the reduce to the following:
You get the correct result. This is really strange. Things like this used to work, and a lot of your template functions in rhoptions() do this type of thing. I don't know if it's something in the internals of R that has changed or what. Here's my session info: