search

EclairJS / eclairjs-nashorn

JavaScript API for Apache Spark
Apache License 2.0
94 stars 11 forks source link

support the equivalent of session.createDataset(data, data.class) #308

Open pberkland opened 8 years ago

pberkland commented 8 years ago

The spark programming style of session.createDataset(javaObject, javaObject.class) is much more common in Spark 2.0. We need some kind of JS equivalent.

We should be support something like createDataset(jsObject, { fieldName : "String", fieldName2:"Integer"});

This will probably require creating a JSON encoder which is not derived from the RowEncoder.

pberkland commented 8 years ago

Doing a real JSON encoder which is not derived from RowEncoder would be difficult, because encoder implementations generate code under the covers. This is done thru internal classes. Trying to do this for JSON would require good understanding of the Encoder internals.

Possible future versions of spark will make it easier to create custom encoders.

The 'createDatasetFromJSON' api seems to be working for current needs.