snappydata on external spark cluster (mapr)

[thbeh]

Hi, trying to run the following and got some "Instruction type does not match stack map" (refer below) [mapr@myspark ~]$ /opt/mapr/spark/spark-2.0.1/bin/spark-shell --master yarn --conf spark.snappydata.store.locators=192.168.100.88:10334 --packages "SnappyDataInc:snappydata:0.7-s_2.11" Ivy Default Cache set to: /home/mapr/.ivy2/cache The jars for the packages stored in: /home/mapr/.ivy2/jars :: loading settings :: url = jar:file:/opt/mapr/spark/spark-2.0.1/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml SnappyDataInc#snappydata added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found SnappyDataInc#snappydata;0.7-s_2.11 in spark-packages :: resolution report :: resolve 264ms :: artifacts dl 4ms :: modules in use: SnappyDataInc#snappydata;0.7-s_2.11 from spark-packages in [default]

|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
|      default     |   1   |   0   |   0   |   0   ||   1   |   0   |
---------------------------------------------------------------------

:: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 1 already retrieved (0kB/9ms) 17/02/24 00:23:38 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 17/02/24 00:24:02 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect. Spark context Web UI available at http://192.168.100.96:4040 Spark context available as 'sc' (master = yarn, app id = application_1485921759427_0016). Spark session available as 'spark'. Welcome to

---

 / __/__  ___ _____/ /__
_\ \/ _ \/ _ `/ __/  '_/

// ./_,// //_\ version 2.0.1-mapr-1611 /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_121) Type in expressions to have them evaluated. Type :help for more information.

scala> import org.apache.spark.sql.{SnappySession, SparkSession} import org.apache.spark.sql.{SnappySession, SparkSession}

scala> val snappy = new org.apache.spark.sql.SnappySession(spark.sparkContext) snappy: org.apache.spark.sql.SnappySession = org.apache.spark.sql.SnappySession@64dcfd09

scala> val colTable = snappy.table("TestColumnTable") java.lang.VerifyError: Instruction type does not match stack map Exception Details: Location: org/apache/spark/sql/SnappyDDLParser.createTable()Lorg/parboiled2/Rule; @1192: aload Reason: Type 'org/apache/spark/sql/execution/datasources/CreateTableUsing' (current frame, locals[53]) is not assignable to 'org/apache/spark/sql/catalyst/plans/logical/LeafNode' (stack map, locals[53]) Current Frame: bci: @1192 ....

[hbhanawat]

SnappyData 0.7 is based on Spark 2.0.2. In Smart connector mode, it can work with Spark 2.0.2 versions only. This short coming would be fixed with soon to be released SnappyData 0.7.1 which will work with Spark 2.0.0 and Spark 2.0.1 as well.

[thbeh]

Hi,

Is there a time line or is that something I can do e.g. recompiling as I am trying to setup for a demo which about 2 weeks from now.

Thanks in advance

Regards Beh

On Tue, Feb 28, 2017 at 9:55 PM, hbhanawat notifications@github.com wrote:

SnappyData 0.7 is based on Spark 2.0.2. In Smart connector mode, it can work with Spark 2.0.2 versions only. This short coming would be fixed with soon to be released SnappyData 0.7.1 which will work with Spark 2.0.0 and Spark 2.0.1 as well.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-282980358, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhnJD9x0LA_yLzgSiSIAlpqpgw7yQks5rg-DkgaJpZM4MKxtD .

[hbhanawat]

We are in the process of testing the release. If you need something for a demo, we can give you a private build with minimal testing. Would that work for you?

In case you are aware of the process of building SnappyData, you can build it from the branch - branch-0.7.1.

[thbeh]

Building on branch-0.7.1 sound like a plan but how would I put in the spark-submit --packages context (if I am correct "--packages "SnappyDataInc:snappydata:0.7-s_2.11" refers to Maven repo).

Please advice?

Regards Beh

On Tue, Feb 28, 2017 at 10:35 PM, hbhanawat notifications@github.com wrote:

We are in the process of testing the release. If you need something for a demo, we can give you a private build with minimal testing. Would that work for you?

In case you are aware of the process of building SnappyData, you can build it from the branch - branch-0.7.1.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-282989646, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhhnOR584BEkZ-xc_rUvNrYyL-_Ajks5rg-pngaJpZM4MKxtD .

[hbhanawat]

Well you can pass a jar on a local file system to spark-submit using --jars command line option.

On Feb 28, 2017 4:15 PM, "T H Beh" notifications@github.com wrote:

Building on branch-0.7.1 sound like a plan but how would I put in the spark-submit --packages context (if I am correct "--packages "SnappyDataInc:snappydata:0.7-s_2.11" refers to Maven repo).

Please advice?

Regards Beh

On Tue, Feb 28, 2017 at 10:35 PM, hbhanawat notifications@github.com wrote:

We are in the process of testing the release. If you need something for a demo, we can give you a private build with minimal testing. Would that work for you?

In case you are aware of the process of building SnappyData, you can build it from the branch - branch-0.7.1.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment- 282989646, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhhnOR584BEkZ-xc_ rUvNrYyL-_Ajks5rg-pngaJpZM4MKxtD .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-283006170, or mute the thread https://github.com/notifications/unsubscribe-auth/AH-OyJNUxpKU_cB0O0IdmgcY7qIr6jZjks5rg_rDgaJpZM4MKxtD .

[Richard-bizruntime]

How to set jar file in snappydata, While running the Snappydata installation i am running snappydataHomeDir$ ./sbin/start-all.sh commend but it not running its showing "main" java.lang.NoClassDefFoundError: org/slf4j/Logger then i added the Jar file in snappydata jars folder even though its not taking the jar how to fix that issue.

[thbeh]

Which specific jar should I include after rebuilding 0.7.1?

Thanks

On Wed, Mar 1, 2017 at 2:42 AM, hbhanawat notifications@github.com wrote:

Well you can pass a jar on a local file system to spark-submit using --jars command line option.

On Feb 28, 2017 4:15 PM, "T H Beh" notifications@github.com wrote:

Building on branch-0.7.1 sound like a plan but how would I put in the spark-submit --packages context (if I am correct "--packages "SnappyDataInc:snappydata:0.7-s_2.11" refers to Maven repo).

Please advice?

Regards Beh

On Tue, Feb 28, 2017 at 10:35 PM, hbhanawat notifications@github.com wrote:

We are in the process of testing the release. If you need something for a demo, we can give you a private build with minimal testing. Would that work for you?

In case you are aware of the process of building SnappyData, you can build it from the branch - branch-0.7.1.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment- 282989646, or mute the thread https://github.com/notifications/unsubscribe- auth/AHBFhhnOR584BEkZ-xc_ rUvNrYyL-_Ajks5rg-pngaJpZM4MKxtD .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment- 283006170, or mute the thread https://github.com/notifications/unsubscribe-auth/AH-OyJNUxpKU_ cB0O0IdmgcY7qIr6jZjks5rg_rDgaJpZM4MKxtD

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-283041822, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhjyzOPzu-lLn7hQTPEdhzDVzK3vzks5rhCQzgaJpZM4MKxtD .

[hbhanawat]

After building, use $SNAPPY_HOME/build-artifacts/scala-2.11/distributions/snappydata-core_2.11-*.jar. This contains the packages to run the Smart connector mode.

On Wed, Mar 1, 2017 at 1:58 AM, T H Beh notifications@github.com wrote:

Which specific jar should I include after rebuilding 0.7.1?

Thanks

On Wed, Mar 1, 2017 at 2:42 AM, hbhanawat notifications@github.com wrote:

Well you can pass a jar on a local file system to spark-submit using --jars command line option.

On Feb 28, 2017 4:15 PM, "T H Beh" notifications@github.com wrote:

Building on branch-0.7.1 sound like a plan but how would I put in the spark-submit --packages context (if I am correct "--packages "SnappyDataInc:snappydata:0.7-s_2.11" refers to Maven repo).

Please advice?

Regards Beh

On Tue, Feb 28, 2017 at 10:35 PM, hbhanawat notifications@github.com wrote:

We are in the process of testing the release. If you need something for a demo, we can give you a private build with minimal testing. Would that work for you?

In case you are aware of the process of building SnappyData, you can build it from the branch - branch-0.7.1.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/ issues/516#issuecomment- 282989646, or mute the thread https://github.com/notifications/unsubscribe- auth/AHBFhhnOR584BEkZ-xc_ rUvNrYyL-_Ajks5rg-pngaJpZM4MKxtD .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment- 283006170, or mute the thread https://github.com/notifications/unsubscribe-auth/AH-OyJNUxpKU_ cB0O0IdmgcY7qIr6jZjks5rg_rDgaJpZM4MKxtD

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment- 283041822, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhjyzOPzu- lLn7hQTPEdhzDVzK3vzks5rhCQzgaJpZM4MKxtD .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-283151721, or mute the thread https://github.com/notifications/unsubscribe-auth/AH-OyNTcvD3f0xqpAl2sSSXQGUHopgjpks5rhIOKgaJpZM4MKxtD .

[hbhanawat]

While running the Snappydata installation i am running snappydataHomeDir$ ./sbin/start-all.sh commend but it not running its showing "main" java.lang.NoClassDefFoundError: org/slf4j/Logger then i added the Jar file in snappydata jars folder even though its not taking the jar how to fix that issue.

start-all command starts the standalone cluster manager for Spark. SnappyData jars are not needed for this. When the an application that connects with a Snappy cluster is launched using spark-submit, the snappydata jars are needed and these jars can be passed using --jars command line option.

[Richard-bizruntime]

I started the spark cluster without any issue, While starting the snappy server ./sbin/snappy-start-all.sh its not starting its showing Could not find or load main class io.snappydata.tools.SnappyUtilLauncher , how to fix that one ,Thanks Advanced.

[hbhanawat]

Did you set the SPARK_HOME as spark-2.0.2-bin-without-hadoop? While running snappy scripts, it looks for SPARK_HOME and it should be set as the snappy installation folder. This is because snappy installation also includes spark installation.

On Wed, Mar 1, 2017 at 11:45 AM, Richard-bizruntime < notifications@github.com> wrote:

I started the spark cluster without any issue, While starting the snappy server ./sbin/snappy-start-all.sh its not starting its showing ../snappydata-0.7-bin/sbin/snappy-locators.sh: line 30: ../spark-2.0.2-bin-without-hadoop/bin/load-snappy-env.sh: No such file or directory i checked my snappy and spark conf folder there is no such a file , how to fix that one ,Thanks Advanced.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-283254820, or mute the thread https://github.com/notifications/unsubscribe-auth/AH-OyKvpaQIypp7vRk18lvnR_v3NAkkLks5rhQ0NgaJpZM4MKxtD .

[Richard-bizruntime]

I export the snappyhome directory$ export SPARK_HOME=/home/bizruntime/hadoop/hadoospark/spark-2.0.2-bin-without-hadoop like that then configure in the snappydata-0.7-bin/conf$spark-env.sh folder also same issue i need to configure any other this thinks

[hbhanawat]

I meant set

export SPARK_HOME=.../snappydata-0.7-bin

Don't set SPARK_HOME as the spark-2.0.2-bin-without-hadoop while running snappy scripts.

[Richard-bizruntime]

Thank you so much..snappy server started successfully,

[Richard-bizruntime]

Hi hbhanawat. I followed by this link (https://github.com/SnappyDataInc/zeppelin-interpreter) as per the snappy-data Documentation connect to the zeppelin to snappy-data, i don't know where i need to configure the lead configuration i configured the conf/lead file but its not working , do you have any idea , Thanks in Advanced.

[hbhanawat]

The conf/leads should have an entry like this.

leadhost -zeppelin.interpreter.enable=true

Is this not working?

Hemant

On Thu, Mar 2, 2017 at 3:24 PM, Richard-bizruntime <notifications@github.com

wrote:

Hi hbhanawat. I followed by this [link] (https://github.com/SnappyDataInc/zeppelin- interpreter) as per the snappy-data Documentation connect to the zeppelin to snappy-data, i don't know where i need to configure the lead configuration i configured the conf/lead file but its not working , do you have any idea , Thanks in Advanced.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-283608146, or mute the thread https://github.com/notifications/unsubscribe-auth/AH-OyE2ix2kvz4Owj5X0sY5aRN-hMJQTks5rhpHBgaJpZM4MKxtD .

[Richard-bizruntime]

I given like localhost zeppelin.interpreter.enable true

[hbhanawat]

Did it work with what I suggested?

On Thu, Mar 2, 2017 at 4:05 PM, Richard-bizruntime <notifications@github.com

wrote:

I given like localhost zeppelin.interpreter.enable true

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-283617730, or mute the thread https://github.com/notifications/unsubscribe-auth/AH-OyMLsQJzDoZfUhSusiJjDKitVKgFYks5rhpuKgaJpZM4MKxtD .

[Richard-bizruntime]

Ok thank you hbhanawat.

[Richard-bizruntime]

@hbhanawat As per your suggestion i added (localhost -zeppelin.interpreter.enable=true) in conf/lead, after i started the zeppelin server,But as per the documentation the snappy-data with zeppelin port 3768 its not running if i use the another server /locator port it will through Can't get RemoteInterpreterEvent org.apache.thrift.transport.TTransportException , so remotely i can't access the snappy-data in zeppelin. could you please tell me what i have to do for snappy data lead configuration thanks in advanced..

[SachinJanani]

Hi @Richard-bizruntime have you change the interpreter settings in zeppelin UI to allow zeppelin server to connect to the snappydata interpreter running on the lead as shown in this image

[Richard-bizruntime]

@SachinJanani yes i changed as per showing in the Image , while running its showing this org.apache.thrift.transport.TTransportException error on zeppelin as well as log file also, could you tell me how to fix that issue thanks in advanced.

[thbeh]

I am still getting the same issues after building branch-7.1. Here's the step I took -

git clone https://github.com/SnappyDataInc/snappydata.git https://github.com/SnappyDataInc/snappydata.git -b branch-0.7.1 --recursive cd snappydata ./gradlew product

The above was taken from snappydata doc - Building from source. After the built I notice the following in the distributions folder -

$ ls -l ~/snappydata/build-artifacts/scala-2.11/distributions/ total 65956 -rw-rw-r--. 1 centos centos 26554 Mar 3 20:45 libgemfirexd64_g.so -rw-rw-r--. 1 centos centos 6938 Mar 3 20:45 libgemfirexd64.so -rw-rw-r--. 1 centos centos 5216253 Mar 2 08:42 snappydata-client-1.5.3.jar -rw-rw-r--. 1 centos centos 62281853 Mar 2 08:42 snappydata-core_2.11-0.7.jar

The 2 lib files were not originally there but when I ran spark-shell it complain that the lib file was missing and when I copied there the complain disappear. When I ran the spark-shell (v2.0.1), I get the following and also started to notice the entrophy message. Please advice. Thanks

[mapr@myspark ~]$ /opt/mapr/spark/spark-2.0.1/bin/spark-shell --master yarn --conf spark.snappydata.store.locators=locator1:10334 --jars /home/mapr/snappydata/build-artifacts/scala-2.11/distributions/snappydata-core_2.11-0.7.jar 17/03/03 20:57:02 WARN ThreadLocalRandom: Failed to generate a seed from SecureRandom within 3 seconds. Not enough entrophy? 17/03/03 20:57:03 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 17/03/03 20:57:32 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect. Spark context Web UI available at http://192.168.100.96:4040 http://192.168.100.96:4040 Spark context available as 'sc' (master = yarn, app id = application_1485921759427_0052). Spark session available as 'spark'. Welcome to

• __ *
• / / _ ___/ /__*
• \ \/ \/ _ `/ _/ '/*
• // ./_,// //_\ version 2.0.1-mapr-1611*
• /_/*

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_121) Type in expressions to have them evaluated. Type :help for more information.

scala> import org.apache.spark.sql.{SnappySession, SparkSession} import org.apache.spark.sql.{SnappySession, SparkSession}

scala> val snSession = new SnappySession(sc) snSession: org.apache.spark.sql.SnappySession = org.apache.spark.sql.SnappySession@2396dc4e

scala> val colTable = snSession.table("TestColumnTable") java.lang.VerifyError: Instruction type does not match stack map Exception Details:

• Location:*

• org/apache/spark/sql/SnappyDDLParser.createTable()Lorg/parboiled2/Rule; @1192: aload*

• Reason:*
• Type 'org/apache/spark/sql/execution/datasources/CreateTableUsing' (current frame, locals[53]) is not assignable to 'org/apache/spark/sql/catalyst/plans/logical/LeafNode' (stack map, locals[53])*
• Current Frame:*

On Wed, Mar 1, 2017 at 5:28 PM, hbhanawat notifications@github.com wrote:

After building, use $SNAPPY_HOME/build-artifacts/scala-2.11/distributions/ snappydata-core_2.11-*.jar. This contains the packages to run the Smart connector mode.

On Wed, Mar 1, 2017 at 1:58 AM, T H Beh notifications@github.com wrote:

Which specific jar should I include after rebuilding 0.7.1?

Thanks

On Wed, Mar 1, 2017 at 2:42 AM, hbhanawat notifications@github.com wrote:

Well you can pass a jar on a local file system to spark-submit using --jars command line option.

On Feb 28, 2017 4:15 PM, "T H Beh" notifications@github.com wrote:

Building on branch-0.7.1 sound like a plan but how would I put in the spark-submit --packages context (if I am correct "--packages "SnappyDataInc:snappydata:0.7-s_2.11" refers to Maven repo).

Please advice?

Regards Beh

On Tue, Feb 28, 2017 at 10:35 PM, hbhanawat < notifications@github.com> wrote:

We are in the process of testing the release. If you need something for a demo, we can give you a private build with minimal testing. Would that work for you?

In case you are aware of the process of building SnappyData, you can build it from the branch - branch-0.7.1.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/ issues/516#issuecomment- 282989646, or mute the thread https://github.com/notifications/unsubscribe- auth/AHBFhhnOR584BEkZ-xc_ rUvNrYyL-_Ajks5rg-pngaJpZM4MKxtD .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/ issues/516#issuecomment- 283006170, or mute the thread https://github.com/notifications/unsubscribe-auth/AH-OyJNUxpKU_ cB0O0IdmgcY7qIr6jZjks5rg_rDgaJpZM4MKxtD

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment- 283041822, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhjyzOPzu- lLn7hQTPEdhzDVzK3vzks5rhCQzgaJpZM4MKxtD .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment- 283151721, or mute the thread https://github.com/notifications/unsubscribe-auth/AH- OyNTcvD3f0xqpAl2sSSXQGUHopgjpks5rhIOKgaJpZM4MKxtD

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-283241793, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhrLl3q7T5jVu-tt_94vo6nqPOqcKks5rhPP3gaJpZM4MKxtD .

[Richard-bizruntime]

How can we use REST API in snappy-data there have any Option for that, we can access snappy-data database with rest full query execution??

[Richard-bizruntime]

How to use snappy-shell script as a Job in Snappy-data Job like a smart connect mode example , we can run only in snappy-shell only in that example , i need to see the Output in spark job server UI , so i need to run that as a snappy shell as a snappy job there have a any solution on it, please share me if its possible thanks in advanced.. My code Example: val snappy = org.apache.spark.sql.SnappyContext(spark.sparkContext) import snappy.implicits._ snappy.sql("drop table if exists mytable") snappy.sql("create table mytable(data srting,data1 srting,data2 srting,data3 srting,) using column") val tableSchema = snappy.table("mytable").schema val customerDF = snappy.read.schema(schema = tableSchema).csv(s"adl://mylake.azuredatalakestore.net/test.csv") customerDF.write.insertInto("mytable") snappy.sql("select data, data2 from mytable").show

[hbhanawat]

@thbeh I could reproduce this issue. Will get back to you with more details.

[hbhanawat]

@thbeh I just pushed a change to 0.7.1 branch. Can you try now and let me know?

[jramnara]

@Richard-bizruntime , can you start a new thread? this thread seems to be discussing an unrelated issue (from thbeh). Anyway, I am 100% clear on your question. Looks like you want to run a SQL job but have this job be visible in the spark UI? if so, you can click the SQL tab and you will see the individual statements - logical, physical plan , etc.

[thbeh]

@hbhanawat, i will have a go and update later. Thanks

On Tue, Mar 7, 2017 at 1:52 AM, hbhanawat notifications@github.com wrote:

@thbeh https://github.com/thbeh I just pushed a change to 0.7.1 branch. Can you try now and let me know?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-284388014, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhrl6sPzMtpzXDLk50J7G4glMutajks5rjAGmgaJpZM4MKxtD .

[thbeh]

Hi hbhanawat,

So the compilation work but I got some complain of hive folder not found (refer towards the end of the attached) -

[mapr@myspark ~]$ /opt/mapr/spark/spark-2.0.1/bin/spark-shell --master yarn --jars '/home/mapr/ext-jars/snappydata-core_2.11-0.7.jar' --conf spark.snappydata.store.locators=locator1:10334 17/03/06 20:27:50 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1070) 17/03/06 20:28:04 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect. Spark context Web UI available at http://192.168.100.96:4040 Spark context available as 'sc' (master = yarn, app id = application_1485921759427_0065). Spark session available as 'spark'. Welcome to

---

 / __/__  ___ _____/ /__
_\ \/ _ \/ _ `/ __/  '_/

// ./_,// //_\ version 2.0.1-mapr-1611 /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_121) Type in expressions to have them evaluated. Type :help for more information.

scala> import org.apache.spark.sql.{SnappySession, SparkSession} import org.apache.spark.sql.{SnappySession, SparkSession}

scala> val snSession = new SnappySession(sc) snSession: org.apache.spark.sql.SnappySession = org.apache.spark.sql.SnappySession@2ae0eb98

scala> val colTable = snSession.table("TestColumnTable") java.lang.RuntimeException: java.io.FileNotFoundException: File /user/mapr/tmp/hive does not exist at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) at org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:189) at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:247) at org.apache.spark.sql.hive.HiveClientUtil.newClient(HiveClientUtil.scala:235) at org.apache.spark.sql.hive.HiveClientUtil.(HiveClientUtil.scala:129) at org.apache.spark.sql.internal.SnappySharedState.metadataHive$lzycompute(SnappySharedState.scala:33) at org.apache.spark.sql.internal.SnappySharedState.metadataHive(SnappySharedState.scala:33) at org.apache.spark.sql.internal.SnappySharedState.externalCatalog$lzycompute(SnappySharedState.scala:37) at org.apache.spark.sql.internal.SnappySharedState.externalCatalog(SnappySharedState.scala:36) at org.apache.spark.sql.internal.SnappySessionState.catalog$lzycompute(SnappySessionState.scala:200) at org.apache.spark.sql.internal.SnappySessionState.catalog(SnappySessionState.scala:199) at org.apache.spark.sql.internal.SnappySessionState.catalog(SnappySessionState.scala:53) at org.apache.spark.sql.SparkSession.table(SparkSession.scala:568) at org.apache.spark.sql.SparkSession.table(SparkSession.scala:564) ... 48 elided Caused by: java.io.FileNotFoundException: File /user/mapr/tmp/hive does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:607) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:877) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:597) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:602) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508) ... 61 more

scala> :q 17/03/06 20:30:55 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(1,WrappedArray()) [mapr@myspark ~]$ hadoop fs -ls /user/mapr/tmp/hive Found 1 items drwxr-xr-x - mapr mapr 1 2017-01-31 04:05 /user/mapr/tmp/hive/mapr [mapr@myspark ~]$ hadoop fs -ls /user/mapr/tmp/hive Found 1 items drwxr-xr-x - mapr mapr 1 2017-01-31 04:05 /user/mapr/tmp/hive/mapr [mapr@myspark ~]$ hadoop fs -ls /user/mapr/tmp/hive/mapr Found 1 items drwxr-xr-x - mapr mapr 1 2017-01-31 04:05 /user/mapr/tmp/hive/mapr/9668005d-1bfe-4ce1-8513-f4a95ac3b8b3 [mapr@myspark ~]$ hadoop fs -ls /user/mapr/tmp/hive/mapr/9668005d-1bfe-4ce1-8513-f4a95ac3b8b3 Found 1 items drwxr-xr-x - mapr mapr 0 2017-01-31 04:05 /user/mapr/tmp/hive/mapr/9668005d-1bfe-4ce1-8513-f4a95ac3b8b3/_tmp_space.db [mapr@myspark ~]$

Cheers Beh

On Tue, Mar 7, 2017 at 7:28 AM, Teik Hooi Beh thbeh@thbeh.com wrote:

@hbhanawat, i will have a go and update later. Thanks

On Tue, Mar 7, 2017 at 1:52 AM, hbhanawat notifications@github.com wrote:

@thbeh https://github.com/thbeh I just pushed a change to 0.7.1 branch. Can you try now and let me know?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/516#issuecomment-284388014, or mute the thread https://github.com/notifications/unsubscribe-auth/AHBFhrl6sPzMtpzXDLk50J7G4glMutajks5rjAGmgaJpZM4MKxtD .

[Richard-bizruntime]

@jramnara ,Thank you, Yes, you are Right, Then i have another query on that, there have any possibility, we can run directly snappy-data query execution using any language in pragmatically, like in java we have jdbc connection we can access data base and execute the Query,and we get the Result , like that there have any specific language support for snappy-data access..

[jramnara]

@Richard-bizruntime My previous response had a typo - I meant "I am NOT 100% clear on your question". I still do not understand your scenario and problem. Are you trying to figure out how to run queries from non-Java languages ?

[Richard-bizruntime]

@jramnara ,yes, then i have another query,Could you tell me how to configure the rowstore snappy-data to apache hadoop ,I followed the link http://rowstore.docs.snappydata.io/docs/disk_storage/persist-hdfs-topics.html but i don't understand this link, its new configuration for me, i don't know Kerberos, keytab thinks,if you don't mind could you guide me..Thank you.