search

EclairJS / eclairjs-nashorn

JavaScript API for Apache Spark
Apache License 2.0
94 stars 11 forks source link

Dataset Lamda functions need to be aware of encoder type #307

Open pberkland opened 8 years ago

pberkland commented 8 years ago

The following will cause a class cast exception (can't cast double to integer) var ds = sparkSession.createDataset([1,2,3], Encoders.INT()); ds =ds.map( function(value) { return value + 1; }, Encoders.INT()); This is because the return value from js defaults to double. The lambda function should be aware of the encoder type, and do the appropriate conversion.

pberkland commented 8 years ago

fixed by commit 1fd2e4a299c9551a3ab2e90e564fbc5c350b2307

billreed63 commented 8 years ago

Reopening this issue until we fix creating lamda functions from Java instead of JavaScript. The code in commit 1fd2e4a works from nashorn but not from notebooks.

billreed63 commented 8 years ago

@pberkland above commit is a temp work around.