search

vericast / spylon-kernel

Jupyter kernel for scala and spark
Other
186 stars 37 forks source link

Cannot get Hive data #61

Open JenniferOH opened 1 year ago

JenniferOH commented 1 year ago

Hi I’m Jennifer and I’m having trouble with getting Hive data using scala kernel. I’m testing exact same code with same hive-site.xml(hive config file) on both spark-shell and jupyterlab spylon-kernel

Here’s my jupyterlab code: image

Here’s my spark-shell code:

image

There weren’t many references, and the ones that I’ve tried are:

(Optional) Configuring Spark for Hive Access - Hortonworks Data Platform https://groups.google.com/g/cloud-dataproc-discuss/c/O5wKJDW9kNQ There’s no hive or spark related logs on JupyterLab. Here’s the logs

[W 2022-09-23 09:47:49.887 LabApp] Could not determine jupyterlab build status without nodejs [I 2022-09-23 09:47:50.393 ServerApp] Kernel started: afa4234d-48ac-4505-b6a0-e3fa220161cd [I 2022-09-23 09:47:50.404 ServerApp] Kernel started: 9404ff88-622f-4ba8-86b1-404d648588fc [MetaKernelApp] ERROR | No such comm target registered: jupyter.widget.control [MetaKernelApp] WARNING | No such comm: 94bab30b-35b1-48bf-bb51-6000d46df671 [MetaKernelApp] ERROR | No such comm target registered: jupyter.widget.control [MetaKernelApp] WARNING | No such comm: 936a8763-707c-4076-8471-7ceed85ccb53 Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). [I 2022-09-23 09:57:52.541 ServerApp] Saving file at /scala-spark/Untitled.ipynb [I 2022-09-23 09:59:52.580 ServerApp] Saving file at /scala-spark/Untitled.ipynb [I 2022-09-23 10:21:55.527 ServerApp] Kernel restarted: 9404ff88-622f-4ba8-86b1-404d648588fc [MetaKernelApp] ERROR | No such comm target registered: jupyter.widget.control [MetaKernelApp] WARNING | No such comm: 6eb29ba4-7dab-4314-ace4-88488935840b Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). I’ve also tried this on Jupyter Notebook, removed the kernel and reinstalled it but it was the same :cry:

What could I’ve been missing? Where should I check? Please help.

Thanks Jennifer

Bidek56 commented 1 year ago

@JenniferOH This code seems to work:

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder.config("spark.sql.legacy.createHiveTableByDefault", "false").master("local[*, 4]").appName("Spark").getOrCreate()

but this repo is dead so I would suggest posting your questions on SO.