How to use rest-api(Livy) to access xsql

[lordk911]

The DOC is too simple about how to use rest api to access xsql.

[lordk911]

Here is a patch to livy to support xsql. which offered by XSQL team , I just made a few changes to fit my environment.

Index: server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala   (revision 3c4eab9273775893f56ec1706c1d2b775340cfa3)
+++ server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala   (date 1568004532274)
@@ -334,10 +334,11 @@

     mergeConfList(livyJars(livyConf, scalaVersion), LivyConf.SPARK_JARS)
     val enableHiveContext = livyConf.getBoolean(LivyConf.ENABLE_HIVE_CONTEXT)
+    val enableXSQL = livyConf.getBoolean(LivyConf.ENABLE_XSQL)
     // pass spark.livy.spark_major_version to driver
     builderProperties.put("spark.livy.spark_major_version", sparkMajorVersion.toString)

-    val confVal = if (enableHiveContext) "hive" else "in-memory"
+    val confVal = if (enableXSQL) "xsql" else if (enableHiveContext) "hive" else "in-memory"
     builderProperties.put("spark.sql.catalogImplementation", confVal)

     if (enableHiveContext) {
Index: server/src/main/scala/org/apache/livy/LivyConf.scala
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- server/src/main/scala/org/apache/livy/LivyConf.scala    (revision 3c4eab9273775893f56ec1706c1d2b775340cfa3)
+++ server/src/main/scala/org/apache/livy/LivyConf.scala    (date 1568004454041)
@@ -55,6 +55,7 @@
   val FILE_UPLOAD_MAX_SIZE = Entry("livy.file.upload.max.size", 100L * 1024 * 1024)
   val LOCAL_FS_WHITELIST = Entry("livy.file.local-dir-whitelist", null)
   val ENABLE_HIVE_CONTEXT = Entry("livy.repl.enable-hive-context", false)
+  val ENABLE_XSQL = Entry("livy.repl.enable-xsql", false)

   val ENVIRONMENT = Entry("livy.environment", "production")

Index: rsc/pom.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- rsc/pom.xml (revision 3c4eab9273775893f56ec1706c1d2b775340cfa3)
+++ rsc/pom.xml (date 1568004648183)
@@ -91,12 +91,14 @@
     </dependency>
     <dependency>
       <groupId>org.apache.spark</groupId>
-      <artifactId>spark-hive_${scala.binary.version}</artifactId>
+      <artifactId>xsql-hive_${scala.binary.version}</artifactId>
+      <version>${xsql.version}</version>
       <scope>provided</scope>
     </dependency>
     <dependency>
       <groupId>org.apache.spark</groupId>
-      <artifactId>spark-sql_${scala.binary.version}</artifactId>
+      <artifactId>xsql-sql_${scala.binary.version}</artifactId>
+      <version>${xsql.version}</version>
       <scope>provided</scope>
     </dependency>
     <dependency>
Index: server/pom.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- server/pom.xml  (revision 3c4eab9273775893f56ec1706c1d2b775340cfa3)
+++ server/pom.xml  (date 1568004623377)
@@ -201,7 +201,8 @@

     <dependency>
       <groupId>org.apache.spark</groupId>
-      <artifactId>spark-sql_${scala.binary.version}</artifactId>
+      <artifactId>xsql-sql_${scala.binary.version}</artifactId>
+      <version>${xsql.version}</version>
       <scope>test</scope>
     </dependency>

@@ -213,7 +214,8 @@

     <dependency>
       <groupId>org.apache.spark</groupId>
-      <artifactId>spark-hive_${scala.binary.version}</artifactId>
+      <artifactId>xsql-hive_${scala.binary.version}</artifactId>
+      <version>${xsql.version}</version>
       <scope>test</scope>
     </dependency>

Index: rsc/src/main/java/org/apache/livy/rsc/driver/SparkEntries.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- rsc/src/main/java/org/apache/livy/rsc/driver/SparkEntries.java  (revision 3c4eab9273775893f56ec1706c1d2b775340cfa3)
+++ rsc/src/main/java/org/apache/livy/rsc/driver/SparkEntries.java  (date 1567762758438)
@@ -80,6 +80,10 @@
               builder.enableHiveSupport();
               sparksession = builder.getOrCreate();
               LOG.info("Created Spark session (with Hive support).");
+            } else if (catalog.equals("xsql") && SparkSession$.MODULE$.xsqlClassesArePresent()) {
+              builder.enableXSQLSupport();
+              sparksession = builder.getOrCreate();
+              LOG.info("Created Spark session (with XSQL support).");
             } else {
               builder.config("spark.sql.catalogImplementation", "in-memory");
               sparksession = builder.getOrCreate();
Index: pom.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- pom.xml (revision 3c4eab9273775893f56ec1706c1d2b775340cfa3)
+++ pom.xml (date 1567768692282)
@@ -82,6 +82,7 @@
     <hadoop.scope>compile</hadoop.scope>
     <spark.scala-2.11.version>2.2.3</spark.scala-2.11.version>
     <spark.version>${spark.scala-2.11.version}</spark.version>
+    <xsql.version>0.7.0-SNAPSHOT</xsql.version>
     <hive.version>3.0.0</hive.version>
     <commons-codec.version>1.9</commons-codec.version>
     <httpclient.version>4.5.3</httpclient.version>

[lordk911]

mvn package -P spark-2.4 -DskipTests

[beliefer]

We have a patch to modify Livy code so as to support XSQL. @WeiWenda Please tell the patch how to use?

[beliefer]

I think we should create some directory to store the Livy patch.

[beliefer]

@lordk911 Do you resolved this issue?