search

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.89k stars 4.27k forks source link

Apache Beam Java Lib and ANTLR #32696

Open pinfrederic opened 1 month ago

pinfrederic commented 1 month ago

I am using Apache Beam with Google Dataflow in Java and I am trying to migrate the java lib from v2.51.0 to v2.59.0

We are using ANTLR v4.11.1 as a dependency of our Beam Pipeline.

With the v2.59.0 the pipeline fails to start with the error ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.11.1

I can see both Apache Beam v2.51.0 and v2.59.0 are compiled with ANTLR v4.7 as a dependency HERE and HERE

My questions are the following: Why do I suddenly get an exception at runtime with v2.59.0 when the v2.51.0 was working correctly? I am not very confident to downgrade our version of ANTLR just because Apache Beam uses the v4.7. What if you upgrade it to v4.13? I would need to adapt again... this problem can be met by a lot of users of Apache Beam, is it possible to get rid of this dependency on Apache Beam lib?

ygw commented 3 weeks ago

We are running into a similar issue where we have a 4.13 dependency elsewhere in our codebase, and now 4.7 through Beam. Can Beam upgrade to 4.13?

Abacn commented 2 weeks ago

I'm trying to understand the use case. Beam Java core should have repackaged antlr core and antlr runtime and shaded in the released jar. Inspect the beam-sdks-java-core jar, org.antlr.v4 classes are directed to org.apache.beam.repackaged.core.org.antlr.v4 namespace.

User program should be able to use newer antlr version. You can exclude antlr in your project, e.g.

implementation('org.apache.beam:beam-sdks-java-core:2.60.0') {
        exclude group: 'org.antlr', module: '*'
    }