apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.77k stars 4.21k forks source link

[Failing Test]: Java Samza runner tests failing on Java11 #32208

Open Abacn opened 4 weeks ago

Abacn commented 4 weeks ago

What happened?

Error message:

org.apache.beam.sdk.Pipeline$PipelineExecutionException: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IncompatibleClassChangeError: Method 'void org.apache.samza.storage.kv.KeyValueStore.deleteAll(java.util.List)' must be InterfaceMethodref constant
    at app//org.apache.beam.runners.samza.SamzaPipelineResult.getStateInfo(SamzaPipelineResult.java:124)
...
Caused by: java.lang.IncompatibleClassChangeError: Method 'void org.apache.samza.storage.kv.KeyValueStore.deleteAll(java.util.List)' must be InterfaceMethodref constant
    at org.apache.samza.storage.kv.CachedStore.deleteAll(CachedStore.scala:253)
    at org.apache.samza.storage.kv.NullSafeKeyValueStore.deleteAll(NullSafeKeyValueStore.scala:72)
    at org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$deleteAll$1.apply$mcV$sp(KeyValueStorageEngine.scala:92)
    at org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$deleteAll$1.apply(KeyValueStorageEngine.scala:89)
    at org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$deleteAll$1.apply(KeyValueStorageEngine.scala:89)
    at org.apache.samza.util.TimerUtil$class.updateTimer(TimerUtil.scala:37)
    at org.apache.samza.storage.kv.KeyValueStorageEngine.updateTimer(KeyValueStorageEngine.scala:40)
    at org.apache.samza.storage.kv.KeyValueStorageEngine.deleteAll(KeyValueStorageEngine.scala:89)
    at org.apache.beam.runners.samza.runtime.SamzaStoreStateInternals$SamzaBagState.clear(SamzaStoreStateInternals.java:510)

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

Abacn commented 4 weeks ago

tests also fails on M1 mac due to incompatible jni

Caused by: java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/wg/hwmcqjwd4zz75mjs0r5z_3f400y2yj/T/librocksdbjni1775545785986054429.jnilib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1837)
    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:79)
    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:57)
    at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:69)
    at org.rocksdb.RocksDB.<clinit>(RocksDB.java:38)
    at org.rocksdb.Options.<clinit>(Options.java:24)
    at org.apache.samza.storage.kv.RocksDbOptionsHelper.options(RocksDbOptionsHelper.java:49)

Found that both are fixed on the latest Samza 1.8

Abacn commented 4 weeks ago

Actually bump the version is not trivial , see https://github.com/apache/beam/pull/32209#discussion_r1719034105