SparkR Integration Prototype

[ghost]

Problem DDS should be integrated with SparkR so that that useR's can use DDS functionality from the SparkR shell.

Literature

• http://mail-archives.us.apache.org/mod_mbox/spark-user/201507.mbox/%3C7584C2C1A1745B4CA9E83C53CF180A960ED958D2@SHSMSX101.ccr.corp.intel.com%3E

Possible Strategy i. Write glue code functions for DDS which take an UUID and get the corresponding table.

ii. Create a R package which

• calls some init() function in DDS to hand over the current SparkContext sc object.

iii. For each visualization function in DDS: Provide a R function (e.g., bar() for bar()), which

• takes an DataFrame or Column object.
• registers this object with SparkR::registerAsTempTable() and computes an UUID.
• calls the DDS function via SparkR:::callJStatic(...) (e.g., SparkR:::callJStatic(className = "de.frosner.dds.core.DDS", methodName = "bar", UUID)).
• unregisters the created TempTable afterwards.
• converts the return value (probably a jobj) to some appropriate R object (e.g., a data.frame).
• calls the corresponding R function (e.g., barplot()) to visualize the results.

[FRosner]

I'm afraid that you will not be able to call de.frosner.dds.core.DDS.bar, because it expects a Scala data frame. You would need to write glue code in Scala for each of these functions that takes the name of the temp table (which is the UUID I guess?).

Also you should remove the temp table afterwards.

[ghost]

Thanks, updated the initial issue posting. Let's try this ...

[ghost]

The longer I think about this, the less sense it makes. I think it is much easier and cleaner to just add some wrapper functions in R around the existing and evolving SparkR functionality if one would have an easy way to visualize results in R.

[FRosner]

I think it is much cleaner to enable a generic way for calling Spark packages from PySpark and SparkR

[ghost]

I agree and googled again but something like this doesn't exist atm. As a senior scala developer, you are welcome to submit a proposal/issue. :) If you agree, I will close this issue for now.

[FRosner]

:+1:

On 22 Dec 2015, at 08:45, Basil Komboz notifications@github.com wrote:

I agree and googled again but something like this doesn't exist atm. As a senior scala developer, you welcome to submit a proposal/issue. :) If you agree, I will close this issue for now.

— Reply to this email directly or view it on GitHub https://github.com/FRosner/spawncamping-dds/issues/271#issuecomment-166543324.