piccolbo
commented
10 years ago Same problem I had with outer joins. I ended up doing a quick job to get all the columns and another one to do an explicit conversion. Workaround is returning list(as.data.frame(...)). Then there is no crash, but you are building a list and you can postprocess the list the way you want. It's not ideal but the way serialization is set up since 3.0 it's unlikely this usage will be supported. This is opinion, but I think supporting it was a stretch from the start, because the idea behind rmr2 distributed data structures is that one is storing or creating a data frame, matrix, vector or list. There's no variable column data frame that I know of in R, so why should it be supported in the distributed version? Yes in 2.x you could do this because there was an implicit rbind_fill kind of operation behind the scenes, but it's not something that was deliberately supported nor I think was documented ever. I am guilty of assuming it myself in implementing outer joins, but two wrongs don't make a right. Besides that, there is the problem of implementing this. When serializing, we need to distribute metadata sparingly because the overhead can be crippling. If the map key has many distinct values, we need the data to be split into small chunks, possibly one row each, so that it can travel to the correct reducer. If we send the metadata (class and colnames) with each chunk, then everything is easy, including reassembling a data frame with variable-column parts. But the space overhead can be 120x, without trying too hard (single integer col, single row groups) and can be made +Inf if you pick very long col names. That made performance unpredictable and caused grief to users. In 3.0 there are data chunks and metadata chunks interspersed with them, but no longer associated 1:1. So the overhead can be kept under control, but the metadata chunks have to be all identical. To relax this constraint, you'd have to add a metadata id to each chunk and send the metadata chunks reliably to each reducer, then re-associate them with the data and build a data frame. It's doable, but it's a project, and a challenging one if you add a requirement to do it with little or no performance penalty. How's the workaround looking for you, OK, ugly, blocker?
jamiefolson
Opened on a report by @jamiefolson
When the reduce function returns different columns for different values, the job completes successfully, but any attempt to work with the results will lead to a fatal error crashing the R session:
This may seem like an odd thing to want to do, but for sequences of operations, it is convenient to construct a result sequentially and return as partial results even when the full result cannot be calculated.