[R4ML-247] remove nrow/ncol from sysml matrix conversion

CODAIT / r4ml

Scalable R for Machine Learning

Apache License 2.0

42 stars 13 forks source link

[R4ML-247] remove nrow/ncol from sysml matrix conversion #36

Closed bdwyer2 closed 7 years ago

bdwyer2 commented 7 years ago

Instead of calculating the number of rows/columns in a SparkDataFrame on the SparkR side we let SystemML handle it
Matrix block size is not passed in so the default can be configured via the SystemML settings
R4ML matrix does not cache by default (since it is no longer needed for accurate count)

This PR decreases the time required to convert a 55 GB r4ml.matrix to a SystemML matrix by 69% Tested with the following configuration: 4 executors / 90 GB executor memory / 40 cores per executor

R4ML-CI commented 7 years ago

Build triggered.

R4ML-CI commented 7 years ago

Unit test failed.

R4ML-CI commented 7 years ago

Build triggered.

R4ML-CI commented 7 years ago

Build success. All unit tests passed.