search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
176 stars 90 forks source link

Public API incompatibility: `NamedEntity` class was accidentally shaded in the bundle #723

Closed wajda closed 11 months ago

wajda commented 12 months ago

Discussed in https://github.com/AbsaOSS/spline-spark-agent/discussions/722

Originally posted by **wajda** July 13, 2023 Hello, I am trying to use a postProcessing filter: ![image](https://github.com/AbsaOSS/spline-spark-agent/assets/1311776/7113f795-ce87-4d3d-ad07-0035592bddde) but I am getting this error: ``` Make sure that type NamedEntity is in your classpath and check for conflicting dependencies with `-Ylog-classpath`. A full rebuild may help if 'LineageDispatcher.class' was compiled against an incompatible version of za.co.absa.commons. ``` I am using this version: spark-3.3-spline-agent-bundle_2.12. and spark 3.3.1 It was working before version 1.0. thanks a lot _Originally posted by @kuhnen in https://github.com/AbsaOSS/spline-spark-agent/discussions/713#discussioncomment-6439398_
wajda commented 12 months ago

The issue was caused by PR #715

See discussion #722 for details.

wajda commented 12 months ago

@kuhnen The shading was done to fix an issue occurred when combining Spline agent with other Spark extensions developed internally in Absa. I now understand that the decision wasn't quite optimal from the perspective of the Spline public API backward compatibility support for the wider community. Unfortunately since Spark doesn't provide any classpath isolation for Spark extensions there is no easy solution how to safely bundle shared libraries and maintain the stable API at the same time. While we are discussing the issue, since that actually was the only change in Spline agent 1.2.1 I guess the quickest fix for you will be to temporarily stick with the agent 1.2.0.

wajda commented 11 months ago

We decided to roll back the change done in #715 (ver. 1.2.x) in favour of #729 (2.0.x) So, the next release 1.2.2 will return to the state compatible with the previous 1.2.0, and the breaking change (in better shape) will be released in the 2.0.0.