Missing Dockerfiles for runtime 13

[jklap]

Please create a branch for these latest runtimes. Thanks!

[evanye]

@jingyimei @alexjinghn

[jklap]

Actually at this point, 13.0 is more important for us :)

So changed the issue from 12.0 & 12.1 to 13.0!

[rondefreitas]

I am also looking for this runtime release.

[mdagost]

I'm also looking for this since I need python 3.10. @xinzhao-db -- it looks like you've been working on support for that with commit 759f05d. Are there any instructions for using that?

[mdagost]

I put together a 13.x dockerfile based on @xinzhao-db 's changes: https://gist.github.com/mdagost/271251c7ece8f5376d65f3c84e37c790

The image is here: https://hub.docker.com/r/mdagost/databricksruntime

It appears to work for me on a cluster with 13.0. YMMV.

[alexiswl]

I put together a 13.x dockerfile based on @xinzhao-db 's changes:

@mdagost I got the following error when using this docker image

Command

%python

catalog = 'prod'
sql(f"USE CATALOG '{catalog}'")

Error Traceback

```pytb Py4JJavaError Traceback (most recent call last) File :1 ----> 1 sql(f"USE CATALOG '{catalog}'") File /databricks/spark/python/pyspark/instrumentation_utils.py:48, in _wrap_function..wrapper(*args, **kwargs) 46 start = time.perf_counter() 47 try: ---> 48 res = func(*args, **kwargs) 49 logger.log_success( 50 module_name, class_name, function_name, time.perf_counter() - start, signature 51 ) 52 return res File /databricks/spark/python/pyspark/sql/context.py:560, in SQLContext.sql(self, sqlQuery) 544 def sql(self, sqlQuery: str) -> DataFrame: 545 """Returns a :class:`DataFrame` representing the result of the given query. 546 547 .. versionadded:: 1.0.0 (...) 558 [Row(f1=1, f2='row1'), Row(f1=2, f2='row2'), Row(f1=3, f2='row3')] 559 """ --> 560 return self.sparkSession.sql(sqlQuery) File /databricks/spark/python/pyspark/instrumentation_utils.py:43, in _wrap_function..wrapper(*args, **kwargs) 39 @functools.wraps(func) 40 def wrapper(*args: Any, **kwargs: Any) -> Any: 41 if hasattr(_local, "logging") and _local.logging: 42 # no need to log since this should be internal call. ---> 43 return func(*args, **kwargs) 44 _local.logging = True 45 try: File /databricks/spark/python/pyspark/sql/session.py:1507, in SparkSession.sql(self, sqlQuery, args, **kwargs) 1505 try: 1506 litArgs = {k: _to_java_column(lit(v)) for k, v in (args or {}).items()} -> 1507 return DataFrame(self._jsparkSession.sql(sqlQuery, litArgs), self) 1508 finally: 1509 if len(kwargs) > 0: File /databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command) -> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"): File /databricks/spark/python/pyspark/errors/exceptions/captured.py:185, in capture_sql_exception..deco(*a, **kw) 183 def deco(*a: Any, **kw: Any) -> Any: 184 try: --> 185 return f(*a, **kw) 186 except Py4JJavaError as e: 187 converted = convert_exception(e.java_exception) File /databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE: --> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value)) Py4JJavaError: An error occurred while calling o350.sql. : java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/HiveException at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) at java.lang.Class.getConstructors(Class.java:1651) at org.apache.spark.sql.hive.client.LocalHiveClient.metastore$lzycompute(LocalHiveClientImpl.scala:68) at org.apache.spark.sql.hive.client.LocalHiveClient.metastore(LocalHiveClientImpl.scala:63) at org.apache.spark.sql.hive.client.PoolingHiveClient.$anonfun$new$1(PoolingHiveClient.scala:143) at org.apache.spark.sql.hive.client.PoolingHiveClient.withHiveClient(PoolingHiveClient.scala:149) at org.apache.spark.sql.hive.client.PoolingHiveClient.(PoolingHiveClient.scala:141) at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:426) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:561) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:382) at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:84) at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:80) at org.apache.spark.sql.hive.HiveExternalCatalog.maybeSynchronized(HiveExternalCatalog.scala:116) at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$1(HiveExternalCatalog.scala:156) at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:377) at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:363) at com.databricks.spark.util.SparkDatabricksProgressReporter$.withStatusCode(ProgressReporter.scala:34) at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:155) at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:327) at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:302) at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:297) at org.apache.spark.sql.internal.SharedState.$anonfun$globalTempViewManager$1(SharedState.scala:386) at org.apache.spark.sql.internal.SharedState.$anonfun$globalTempViewExternalCatalogNameCheck$1(SharedState.scala:358) at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) at scala.util.Try$.apply(Try.scala:213) at org.apache.spark.sql.internal.SharedState.globalTempViewExternalCatalogNameCheck(SharedState.scala:358) at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:386) at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:382) at org.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$hiveCatalog$2(HiveSessionStateBuilder.scala:76) at org.apache.spark.sql.catalyst.catalog.SessionCatalogImpl.globalTempViewManager$lzycompute(SessionCatalog.scala:576) at org.apache.spark.sql.catalyst.catalog.SessionCatalogImpl.globalTempViewManager(SessionCatalog.scala:576) at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.globalTempViewManagerName$lzycompute(ManagedCatalogSessionCatalog.scala:109) at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.globalTempViewManagerName(ManagedCatalogSessionCatalog.scala:108) at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.setCurrentDatabaseWithoutCheck(ManagedCatalogSessionCatalog.scala:489) at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.setCurrentCatalog(ManagedCatalogSessionCatalog.scala:348) at com.databricks.sql.DatabricksCatalogManager.setCurrentCatalog(DatabricksCatalogManager.scala:135) at org.apache.spark.sql.execution.command.SetCatalogCommand.run(SetCatalogCommand.scala:30) at org.apache.spark.sql.execution.command.ExecutedCommandExec.$anonfun$sideEffectResult$1(commands.scala:82) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:79) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:91) at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$3(QueryExecution.scala:267) at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:165) at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$2(QueryExecution.scala:267) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$8(SQLExecution.scala:230) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:429) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:175) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1038) at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:125) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:379) at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$1(QueryExecution.scala:266) at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$withMVTagsIfNecessary(QueryExecution.scala:244) at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:262) at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:250) at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:519) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:106) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:519) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:33) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:316) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:312) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:33) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:33) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:495) at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:250) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:372) at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:250) at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:204) at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:195) at org.apache.spark.sql.Dataset.(Dataset.scala:254) at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:123) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1038) at org.apache.spark.sql.SparkSession.$anonfun$withActiveAndFrameProfiler$1(SparkSession.scala:1045) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.SparkSession.withActiveAndFrameProfiler(SparkSession.scala:1045) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:113) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:841) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1038) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:829) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:864) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397) at py4j.Gateway.invoke(Gateway.java:306) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195) at py4j.ClientServerConnection.run(ClientServerConnection.java:115) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.metadata.HiveException at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:358) at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:323) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 93 more ```

[mdagost]

No idea, but the sql function works for me in a notebook:

display(sql(f"SELECT * FROM {table}"))

[evanye]

It might depend on what language type your notebook is. python or scala or sql?

[mdagost]

Given that @alexiswl had an f-string in his code, I assumed it was python. My snippet that works is a python notebook.

[alexiswl]

Given that @alexiswl had an f-string in his code, I assumed it was python. My snippet that works is a python notebook.

Yes was python, have updated to be clearer. I was also running as a job, not sure if that makes a difference.

[DaanRademaker]

I am getting similar error as alexiswl, also running as a job. Did you find a solution @alexiswl ?

[DaanRademaker]

If I try it on a 13.0 cluster, everything works fine

[alexiswl]

Yes I think that may have been my issue and I was originally running on a 12 LTS cluster. I just ended up using the 12 LTS instead of 13.

[DaanRademaker]

In an attempt to get someone at Databricks to answer this issue, I have opened up a discussion here as well: https://community.databricks.com/t5/community-discussions/cannot-use-docker-container-with-runtime-13-1-and-above/m-p/39085#M1073

For some reason 13.1 and 13.2 runtimes do not support Dockerimages the way they used to work.

[DaanRademaker]

For me this issue has been fixed by the way in the 13.3LTS release.

[xinzhao-db]

Close, we publish changes when releasing DBR LTS version