apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
823 stars 163 forks source link

chore: Improve error handling when native lib fails to load #1000

Closed andygrove closed 4 weeks ago

andygrove commented 1 month ago

Which issue does this PR close?

Closes https://github.com/apache/datafusion-comet/issues/999

Rationale for this change

If the static initialization block failed to load the library then we did not see the reason why (in some cases).

Before:

│ 24/10/07 22:42:11 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark                                                                                                   │
│ java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.NativeBase  

After:

│ 24/10/22 15:00:08 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark                                                                                                          │
│ java.lang.UnsatisfiedLinkError: /tmp/libcomet-11952721851997920414.so: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /tmp/libcomet-11952721851997920414.so)      

What changes are included in this PR?

Improve error handling by consuming exceptions in the static init code and then rethrowing them when isCometEnabled callsisLibraryLoaded

How are these changes tested?

Manually.

andygrove commented 1 month ago

@parthchandra could you review?

andygrove commented 1 month ago

@parthchandra @viirya I reimplemented this with a much simpler approach. PTAL when you can.