[BUG] Build fails due to transitive dependencies

dbwiddis commented 1 year ago

What is the bug?

Following merge of https://github.com/opensearch-project/OpenSearch/pull/8434, transitive dependencies between OpenSearch and opensearch-java conflict for Apache Http dependencies.

Execution failed for task ':compileJava'.
> Could not resolve all dependencies for configuration ':compileClasspath'.
   > Conflict(s) found for the following module(s):
       - org.apache.httpcomponents.client5:httpclient5 between versions 5.2.1 and 5.1.4
       - org.apache.httpcomponents.core5:httpcore5 between versions 5.2.2, 5.1.5 and 5.2
       - org.apache.httpcomponents.core5:httpcore5-h2 between versions 5.2.2, 5.1.5 and 5.2

How can one reproduce the bug?

Build latest SDK against latest Opensearch and OpenSearch Client for Java

What is the expected behavior?

Code compiles

Do you have any additional context?

We need to either exclude the transitive dependency or force our own version.

owaiskazi19 commented 1 year ago

This should be fixed by: https://github.com/opensearch-project/opensearch-java/pull/554

dbwiddis commented 1 year ago

Yeah, and we'll break again next time one updates before the other.

cwperks commented 1 year ago

I'm still getting this issue even after the merge of the change in opensearch-java and publishing the latest changes to maven local.

> ./gradlew :dependencyInsight --configuration compileClasspath --dependency org.apache.httpcomponents.core5:httpcore5

org.apache.httpcomponents.core5:httpcore5:5.2.2
+--- org.apache.httpcomponents.core5:httpcore5-h2:5.2.2
|    +--- org.opensearch.client:opensearch-rest-client:3.0.0-SNAPSHOT
|    |    +--- compileClasspath
|    |    \--- org.opensearch.client:opensearch-rest-high-level-client:3.0.0-SNAPSHOT
|    |         \--- compileClasspath
|    +--- org.opensearch.client:opensearch-java:3.0.0-SNAPSHOT
|    |    \--- compileClasspath
|    \--- org.apache.httpcomponents.client5:httpclient5:5.2.1 (requested org.apache.httpcomponents.core5:httpcore5-h2:5.2)
|         +--- org.opensearch.client:opensearch-rest-client:3.0.0-SNAPSHOT (*)
|         \--- org.opensearch.client:opensearch-java:3.0.0-SNAPSHOT (*)
+--- org.opensearch.client:opensearch-java:3.0.0-SNAPSHOT (*)
\--- org.opensearch.client:opensearch-rest-client:3.0.0-SNAPSHOT (*)

I think the most pertinent line is org.apache.httpcomponents.client5:httpclient5:5.2.1 (requested org.apache.httpcomponents.core5:httpcore5-h2:5.2).

httpclient5:5.2.1 is requesting httpcore5-h2:5.2 which mismatches with 5.2.2 being requested elsewhere.

5.2.1 is the latest on maven central: https://mvnrepository.com/artifact/org.apache.httpcomponents.client5/httpclient5

cwperks commented 1 year ago

I had to downgrade httpcore5 to 5.2 in both core and opensearch-java to resolve this issue locally.

reta commented 1 year ago

@cwperks @dbwiddis I will submit the pull request shorlty

dbwiddis commented 1 year ago

\<insert my recurring request for an OpenSearch version catalog here>

reta commented 1 year ago

Hehe .... I hear you, may be needs prioritization ,,,

dbwiddis commented 1 year ago

My background in dependency management is with Maven which seems a bit simpler but easier to resolve situations like this (e.g., order matters). Gradle has a lot of hidden switches and priorities that make these harder to diagnose!

I do think for known transitive dependencies that will have recurring conflicts on version updates, a better solution for SDK is to exclude the transitives from one option (e.g., the client) and thus always use the version in the other (e.g, OpenSearch).

Forcing our own version would work but for transitives seems to break some sort of encapsulation concept.

opensearch-project / opensearch-sdk-java