AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
176 stars 90 forks source link

Decouple from Absa Commons and other external dependencies #729

Closed wajda closed 11 months ago

wajda commented 11 months ago

This is a long-term solution for various classpath conflicts when using Spline agent in combination with other Spark extensions or applications. Shading dependencies help to some extent, but not always, and especially when external classes appear in the agent public API (e.g. #723). To minimize such problems in the future we need:

  1. Make sure the agent public or even developer API do not contain any reference to external classes or interfaces.
  2. Minimize number of external dependencies as much as possible. Copy/paste of some small commons or utility classes and make it a part of our own code base is not necessarily something to be always ashamed of. Everything has its own pros and cons.