Closed IgnasD closed 8 months ago
@IgnasD awesome! Thanks a lot.
Flink is using at least two class loaders to my knowledge: system classloader to load Flink itself and user classloader to load any user JARs as part of his Flink job. I guess the issue happens in session cluster mode, because a scala case class from a checkpoint is loaded by parent class loader, but the same class is already loaded by the user classloader. I think that class equality check is indeed not proper way to check whether classes are compatible. Perhaps proper way to check that is to use Scala own tooling (scala-reflect, etc.). Anyway, I am fine to apply this change to make it work.
In Flink session-mode clusters,
resolveSchemaCompatibility
returnsINCOMPATIBLE
even for identical case classes (even without any recompiles or reuploads to the cluster). This is due to way Flink state restore works, resulting in the same class being loaded multiple times in different classloaders.Steps to reproduce:
To mitigate the issue, I've changed the outer check from direct class equality, which in reality is just reference equality, to class name comparison.