Run the demo SmartConnectorExample.scala with exception table not found

[cherishDWJ]

Exception in thread "main" org.apache.spark.sql.TableNotFoundException: Table 'APP.SNAPPY_COL_TABLE' not found; at org.apache.spark.sql.hive.SnappyStoreHiveCatalog$$anonfun$4.apply(SnappyStoreHiveCatalog.scala:439) at org.apache.spark.sql.hive.SnappyStoreHiveCatalog$$anonfun$4.apply(SnappyStoreHiveCatalog.scala:439) at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) at scala.collection.AbstractMap.getOrElse(Map.scala:59) at org.apache.spark.sql.hive.SnappyStoreHiveCatalog.lookupRelation(SnappyStoreHiveCatalog.scala:438) at org.apache.spark.sql.hive.SnappyStoreHiveCatalog.lookupRelation(SnappyStoreHiveCatalog.scala:475) at org.apache.spark.sql.SparkSession.table(SparkSession.scala:568) at org.apache.spark.sql.SparkSession.table(SparkSession.scala:564) at Snappy.SparkSql.SnappyTest$.main(SnappyTest.scala:22) at Snappy.SparkSql.SnappyTest.main(SnappyTest.scala)

[dshirish]

You need to create table 'APP.SNAPPY_COL_TABLE' as mentioned in the instructions in the comment at the at the top of the file SmartConnectorExample.scala (https://github.com/SnappyDataInc/snappydata/blob/branch-0.8/examples/src/main/scala/org/apache/spark/examples/snappydata/SmartConnectorExample.scala)

[cherishDWJ]

I have created the table, and I can query the data at the command line

[dshirish]

Ok. Is this error seen in your test program? Could you also tell the version you are using?

If you are using 0.8, you will need to use config variable "snappydata.store.locators" while configuring SnappySession. On master, use param 'snappydata.connection' as shown in https://github.com/SnappyDataInc/snappydata/blob/master/examples/src/main/scala/org/apache/spark/examples/snappydata/SmartConnectorExample.scala

[cherishDWJ]

My version is 0.8,I installed single host，and my code use param "snappydata.connection", "192.168.15.121:1527"

[dshirish]

On 0.8 release, use "snappydata.store.locators" config parameter with value as "locatorHost:peer discovery port" for example "localhost:10334" (default port is 10334). "snappydata.connection" param is added on the master branch and will not work on 0.8 version. Refer to the 0.8 example that shows how to instantiate SnappySession at following link: https://github.com/SnappyDataInc/snappydata/blob/branch-0.8/examples/src/main/scala/org/apache/spark/examples/snappydata/SmartConnectorExample.scala

[cherishDWJ]

When I create the table in snappy shell, run the code like this:

 val colTable = snSession.table("SNAPPY_COL_TABLE")
colTable.show(10)

throw exception:

Exception in thread "main" org.apache.spark.sql.TableNotFoundException: Table 'APP.SNAPPY_COL_TABLE' not found;
    at org.apache.spark.sql.hive.SnappyStoreHiveCatalog$$anonfun$4.apply(SnappyStoreHiveCatalog.scala:439)
    at org.apache.spark.sql.hive.SnappyStoreHiveCatalog$$anonfun$4.apply(SnappyStoreHiveCatalog.scala:439)
    at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
    at scala.collection.AbstractMap.getOrElse(Map.scala:59)
    at org.apache.spark.sql.hive.SnappyStoreHiveCatalog.lookupRelation(SnappyStoreHiveCatalog.scala:438)
    at org.apache.spark.sql.hive.SnappyStoreHiveCatalog.lookupRelation(SnappyStoreHiveCatalog.scala:475)
    at org.apache.spark.sql.SparkSession.table(SparkSession.scala:568)
    at org.apache.spark.sql.SparkSession.table(SparkSession.scala:564)
    at Snappy.SparkSql.SmartConnectorExample$.main(SmartConnectorExample.scala:92)
    at Snappy.SparkSql.SmartConnectorExample.main(SmartConnectorExample.scala)

run this code it will be successful:

snSession.dropTable("TestColumnTable", ifExists = true)

    // Creating a table from a DataFrame
    val dataFrame = snSession.range(1000).selectExpr("id", "floor(rand() * 10000) as k")

    snSession.sql("create table TestColumnTable (id bigint not null, k bigint not null) using column")

    dataFrame.write.insertInto("TestColumnTable")

    println(" ####  Write to table completed. ### \n\n" +
        "Now you can query table TestColumnTable using $SNAPPY_HOME/bin/snappy-shell")

     val colTable = snSession.table("SNAPPY_COL_TABLE")
    colTable.show(10)

the result:

 ####  Write to table completed. ### 

Now you can query table TestColumnTable using $SNAPPY_HOME/bin/snappy-shell
+---+----+
| ID|   K|
+---+----+
|471|7028|
|506|4312|
| 85|8484|
|432|6243|
|347|2979|
|183|1022|
|914|2675|
|545|6770|
|682|5028|
|905|5584|
+---+----+
only showing top 10 rows

please tell me why？

I can find the table "TestColumnTable" in snappy shell.

[sumwale]

@diwenjing As @dshirish noted, you are probably using the new "snappydata.connection" property not available in 0.8 release so the spark-shell is creating a single node SnappyData cluster instead of joining to the existing one. Use the "snappydata.store.locators" property on spark-shell start to connect to existing cluster. For example if the locator is running on default peer discovery port of 10334 on the same machines then start spark-shell with: spark-shell --conf snappydata.store.locators=localhost:10334 ...

[sumwale]

@diwenjing Please update with suggestion in previous comments. Please update to latest release 1.0 RC to check "snappydata.connection" property (0.8 and older releases need "snappydata.store.locators"). The final 1.0 release will be out later this week.

[hbf0803]

I got the above problem with running examples in yarn cluster?

I run spark-shell and snappy-sql , and can get table data and description, while run spark-submit on yarn cluster with SnappyData Smart Connector Mode as example : https://snappydatainc.github.io/snappydata/affinity_modes/connector_mode/, get the same problem (table no found). Who can tell me why? (version SnappyData version 1.0.2).

Exception as follow: 18/09/02 16:00:39 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is /home/*/snappydata/snappydata-1.0.2-bin/./warehouse Exception in thread "main" org.apache.spark.sql.TableNotFoundException: Table 'APP.EMPLOYER_TEST' not found; at org.apache.spark.sql.hive.SnappyStoreHiveCatalog.lookupRelation(SnappyStoreHiveCatalog.scala:682) at org.apache.spark.sql.hive.SnappyStoreHiveCatalog.lookupRelation(SnappyStoreHiveCatalog.scala:701) at org.apache.spark.sql.SparkSession.table(SparkSession.scala:578) at org.apache.spark.sql.SparkSession.table(SparkSession.scala:574) at com.test.SnappyDataTest.SnappyWithYarnTest$.main(SnappyWithYarnTest.scala:30) at com.test.SnappyDataTest.SnappyWithYarnTest.main(SnappyWithYarnTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 18/09/02 16:00:39 INFO spark.SparkContext: Invoking stop() from shutdown hook

Job submit: ./bin/spark-submit --master yarn \ --num-executors 3 \ --executor-memory 1g \ --driver-memory 1g \ --conf spark.snappydata.connection=hb-7-centos190:1527 \ --conf spark.driver.extraClassPath=jars/ \ --conf spark.executor.extraClassPath=jars/ \ --class com.test.SnappyWithYarnTest snappydata-test.jar HelloSnappyData

code example: def main(args: Array[String]) { val appName = args(0) val sparkConf = SparkUtil.setSparkConf().setAppName(appName) val sparkSession = SparkSession .builder .appName("SparkApp") .getOrCreate val snappySession = new SnappySession(sparkSession.sparkContext) snappySession.table("APP.EMPLOYER_TEST").collect().foreach(x => println(x.toString())) }

@sumwale @dhavalstr