search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
823 stars 163 forks source link

java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.package$ #1012

Closed andygrove closed 1 month ago

andygrove commented 1 month ago

Describe the bug

I am building a Docker image using the Dockerfile from this repo and then trying to execute queries in k8s. I get the following error.

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.package$
  at org.apache.comet.vector.NativeUtil.<init>(NativeUtil.scala:48)
  at org.apache.comet.CometExecIterator.<init>(CometExecIterator.scala:52)

As far as I can tell, the root cause is an exception being thrown in the following code:

package object comet {

  val CometArrowAllocator = new RootAllocator(Long.MaxValue)

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

tbar4 commented 1 month ago

Getting a similar error:

scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test")
24/10/11 19:32:10 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark
java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.NativeBase
andygrove commented 1 month ago

Getting a similar error:

scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test")
24/10/11 19:32:10 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark
java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.NativeBase

There is a specific issue for the NativeBase error: https://github.com/apache/datafusion-comet/issues/999