Add Flink read/write operations to FlinkExecutionContext

stratosphere / incubator-systemml

Mirror of Apache SystemML (Incubating)

Apache License 2.0

1 stars 4 forks source link

Add Flink read/write operations to FlinkExecutionContext #17

Closed fschueler closed 8 years ago

fschueler commented 8 years ago

In order to add all functionality to the (CSV)ReblockFLInstructions we need at least writeDataSetToHDFS(...) and toMatrixBlock(...) methods in FlinkExecutionContext (see SparkExecutionContext methods for reference).

These are used in MatrixObject.readMatrixFromDataSet(...) which is used by MatrixObject.aquireRead().

fschueler commented 8 years ago

[x] toMatrixBlock(DataSetObject dso, int rlen, int clen, int brlen, int bclen, long nnz)
[x] toMatrixBlock(DataSet<Tuple2<MatrixIndexes, MatrixBlock>> ds, int rlen, int clen, int brlen, int bclen, long nnz)
[x] writeDataSetToHDFS(DataSetObject dso, String path, OutputInfo oinfo)

FelixNeutatz commented 8 years ago

I implemented writeDataSetToHDFS for you since it is really similar to what I did in writing BinaryBlocks in WriteFLInstruction