search

databricks / containers

Sample base images for Databricks Container Services
Apache License 2.0
167 stars 118 forks source link

Custom Jars - Databricks Docker Cluster #120

Open aman-solanki-kr opened 1 year ago

aman-solanki-kr commented 1 year ago

BASE IMAGE - databricksruntime/python:10.4-LTS

I successfully installed the Python dependencies, and the tasks that depend on python in the workflow run fine, but I’m struggling to install the Maven and Jar dependencies.

The jar files are in the docker image (databricks/jars) and are visible in the spark environment path when the cluster starts, but when I trigger the workflow, I see a “Java Package not callable error” since the script is unable to use the classes in the jar files.

aman-solanki-kr commented 1 year ago

@evanye

evanye commented 1 year ago

@aman-solanki-kr Try filing a support ticket with your support rep. Unfortunately I don't know the answer to this.

Nicbyte commented 8 months ago

Just add your jars to /databricks/python3/lib/python3.10/site-packages/pyspark/jars. This is the location for pyspark jars.