search

InitialDLab / Simba

Spatial In-Memory Big data Analytics
Apache License 2.0
120 stars 62 forks source link

Using Simba SQL expressions on Spark 2.1 #92

Open gillesgerlo opened 6 years ago

gillesgerlo commented 6 years ago

Hi guys,

Am I correct in understanding that it's currently not possible to use Simba SQL mode on a Spark 2.1 cluster? I'm much more comfortable using Spark SQL than scala, so would love to hear how you guys are going in migrating this feature.

Thanks!

dongx-psu commented 6 years ago

Yeah, the SQL mode is not available on the standalone version. It is basically because we need to extend the SQL parser to support that and the new antlr based parser is not quite extensible.

Anyway, I will try to find another way around when I have time (maybe make a separate parser path in Simba rather than reuse the parser in Spark SQL).

Thanks for your interest to our project.

Dong On 5/30/2018 8:36:04 AM, Gilles notifications@github.com wrote: Hi guys, Am I correct in understanding that it's currently not possible to use Simba SQL mode on a Spark 2.1 cluster? I'm much more comfortable using Spark SQL than scala, so would love to hear how you guys are going in migrating this feature. Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub [https://github.com/InitialDLab/Simba/issues/92], or mute the thread [https://github.com/notifications/unsubscribe-auth/AAu4JgrkIskHGfDVGVFeeG7yOaJoiqrPks5t3rxhgaJpZM4UThr3].

gillesgerlo commented 6 years ago

Thanks, I appreciate that! I'll start experimenting with the example code you've put up in the mean time. Is any overview available of all functions with their purpose and required parameters? Cheers, Gilles

dongx-psu commented 6 years ago

Sorry for late response. I think the best way is to refer the DataFrame or DataSet API. I believe most of them are already in the examples located in the example source folder.