search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
183 stars 93 forks source link

Unwanted dependencies included in Agent bundles #595

Closed cerveada closed 1 year ago

cerveada commented 1 year ago

Between version 1.0.1 and 1.0.2 Agent bundles grown significantly in size, something is causing lot of unwanted dependencies to be added.

wajda commented 1 year ago

Potentially indirectly causing #587

cerveada commented 1 year ago

The issue was triggered by importing Spark BOM inside of <dependencyManagement> of Agent root pom. As result some <dependencyManagement> elements in bundle-3.3 pom are overwritten by the ones in the root pom.

For example spark-core is not defined as provided (in the effective pom).

This leads to inclusion of several libraries in the bundles that should not supposed to be there.

Another find in the bundles

<!-- this will be overriden -->
<artifactId>spark-core_2.12</artifactId>

<!-- this resolves properly -->
<artifactId>spark-core_${scala.binary.version}</artifactId>

This doesn't make any sense. It is probably a maven bug. I guess some comparision are made without the variable replacement and some with.