search

RevolutionAnalytics / RHadoop

RHadoop
https://github.com/RevolutionAnalytics/RHadoop/wiki
763 stars 278 forks source link

Inconsistent columns cause fatal crash. #215

Closed jamiefolson closed 10 years ago

jamiefolson commented 10 years ago

When the reduce function returns different columns for different values, the job completes successfully, but any attempt to work with the results will lead to a fatal error crashing the R session:

from.dfs(mapreduce(to.dfs(letters),map=function(k,v)keyval(v,v),reduce=function(k,vv)keyval(k,as.data.frame(structure(list(vv),names=vv)))))
...
Segmentation fault.
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/library/RevoScaleR/rxLibs/x64/libExaCore.so.2(_Z21CriticalSignalHandleri+0x1e)[0x7ffe57d4459e]
/lib64/libc.so.6[0x30f9832a20]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0x4d0c4)[0x7ffe6b9a40c4]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0x52da5)[0x7ffe6b9a9da5]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0xbfb1e)[0x7ffe6ba16b1e]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_eval+0x178)[0x7ffe6ba1f6d8]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_applyClosure+0x371)[0x7ffe6ba11fc1]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_eval+0x225)[0x7ffe6ba1f785]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0xc8df7)[0x7ffe6ba1fdf7]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0xc8f80)[0x7ffe6ba1ff80]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0xc425c)[0x7ffe6ba1b25c]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_eval+0x178)[0x7ffe6ba1f6d8]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0xc8df7)[0x7ffe6ba1fdf7]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0xc8f80)[0x7ffe6ba1ff80]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0xc425c)[0x7ffe6ba1b25c]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_eval+0x178)[0x7ffe6ba1f6d8]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_applyClosure+0x371)[0x7ffe6ba11fc1]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0xc5572)[0x7ffe6ba1c572]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_eval+0x178)[0x7ffe6ba1f6d8]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_applyClosure+0x371)[0x7ffe6ba11fc1]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_eval+0x225)[0x7ffe6ba1f785]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(+0x53edb)[0x7ffe6b9aaedb]
/usr/lib64/Revo-7.2/R-3.0.3/lib64/R/lib/libR.so(Rf_eval+0x4dc)[0x7ffe6ba1fa3c]

This may seem like an odd thing to want to do, but for sequences of operations, it is convenient to construct a result sequentially and return as partial results even when the full result cannot be calculated.

piccolbo commented 10 years ago

Hi Jamie, sorry we don't accept rmr2 issues here anymore. I opened RevolutionAnalytics/rmr2#142 on your behalf

jamiefolson commented 10 years ago

Thanks, I didn't even notice I wasn't on the rmr2 page.