search

flink-extended / flink-scala-api

Flink Scala API is a thin wrapper on top of Flink Java API which support Scala Types for serialisation as well the latest Scala version
Apache License 2.0
67 stars 14 forks source link

Scala products are used but Scala API is not on the classpath #177

Open xandril opened 1 week ago

xandril commented 1 week ago

Hello! I am not entirely sure if this relates to the flink-scala-api, but I don't know who else can help with the interaction between Flink and Scala. Currently, I am trying to integrate flink-scala-api using Scala 2.13. As a base image, I am using https://hub.docker.com/layers/apache/flink/1.20-scala_2.12-java8/images/sha256-37750cd7c7f1f33f295cb9415393276b343c72200860863024973466b54cac03?context=explore - this Docker image. In my own image, I remove flink-scala2.12-1.20.0.jar and add scala-library-2.13.15.jar in lib directory, then in the configuration classloader.parent-first-patterns.default, I remove 'scala.' As a result, when attempting to submit a jar with Scala code, I get an exception with the message: "Scala products are used but Scala API is not on the classpath."

Can you suggest what I might be missing in this scenario? @novakov-alexey

novakov-alexey commented 1 week ago

@xandril I think this is because there is no flink-scala-api in the Flink classpath. The error probably comes from here: https://github.com/apache/flink/blob/release-1.20/flink-streaming-java/src/main/java/org/apache/flink/streaming/util/typeutils/FieldAccessorFactory.java#L100

Did you add this library into your job classpath? If yes, the error "Scala products are used but Scala API is not on the classpath" should disappear then.