search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.6k stars 1.01k forks source link

Build system should sanity check transative 3rd party dependencies [LUCENE-5442] #6505

Closed asfimport closed 10 years ago

asfimport commented 10 years ago

SOLR-5365 is an example of a bug that croped up because we upgraded a 3rd party dep (tika) w/o realizing that the version we upgraded too depended on a newer version of another 3rd party dep (commons-compress)

in a comment in SOLR-5365, Jan suggested that it would be nice if there was an easy way to spot problems like this ... i asked steve about it, thinking maybe this is something the maven build could help with, and he mentioned that there is already an ant task to inspect the ivy transative deps in order to generate the maven deps and it could be used to help detect this sort of problem.

opening this issue per steve's request as a reminder to look into this possibility.

Migrated from LUCENE-5442 by Chris M. Hostetter (@hossman), resolved May 30 2014 Attachments: LUCENE-5442.patch Linked issues:

asfimport commented 10 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

We have similar issues with the morphlines stuff (which also heavily uses tika). It's tough because it's extremely hard to have tests that can exercise all of this or to know that you have exercised it all.

It would be great if you could run something that figured out all the maven transitive version dependencies and let us know what version we are behind on or missing.

You still have to harmonize, so it won't necessarily be foolproof, but a great check.

asfimport commented 10 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

It would be great if you could run something that figured out all the maven transitive version dependencies and let us know what version we are behind on or missing.

Yes, and with Tika specifically we have decided to not include all the possible Parser dependencies jars with Solr, so getting such a report will let us do a qualified decision whether to fix the ivy deps or whether to e.g. change the TikaConfig to exclude certain Parsers from the default setup.

asfimport commented 10 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

I've attached a trunk patch that augments the check-lib-versions target to fail the build if a transitive dependency's version is more recent than the corresponding direct dependency's version specified in lucene/ivy-versions.properties. Exceptions are specifiable in a new file lucene/ivy-ignore-conflicts.properties. I've populated this file with the current set of conflicts.

When I comment out the entries in lucene/ivy-ignore-conflicts.properties, this is the output (and the build fails):

[libversions] VERSION CONFLICT: transitive dependency in module(s) benchmark, extraction:
[libversions] /org.apache.commons/commons-compress=1.7
[libversions] +-- /org.tukaani/xz=1.4 <<< Conflict (direct=1.2)
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
[libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
[libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
[libversions]     +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0
[libversions]         +-- /com.sun.jersey/jersey-json=1.9 <<< Conflict (direct=1.8)
[libversions] ... and 15 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
[libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
[libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
[libversions]     +-- /com.sun.jersey/jersey-server=1.9
[libversions]         +-- /asm/asm=3.2 <<< Conflict (direct=3.1)
[libversions] ... and 23 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core:
[libversions] /org.apache.hadoop/hadoop-yarn-server-tests=2.2.0
[libversions] +-- /io.netty/netty=3.6.2.Final
[libversions]     +-- /javax.activation/activation=1.1.1 <<< Conflict (direct=1.1)
[libversions] ... and 14 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
[libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
[libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
[libversions]     +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0
[libversions]         +-- /com.sun.jersey.contribs/jersey-guice=1.9 <<< Conflict (direct=1.8)
[libversions] ... and 13 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) solrj, replicator:
[libversions] /org.apache.httpcomponents/httpclient=4.3.1
[libversions] +-- /commons-logging/commons-logging=1.1.3 <<< Conflict (direct=1.1.1)
[libversions] ... and 1 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) uima:
[libversions] /org.apache.uima/AlchemyAPIAnnotator=2.3.1
[libversions] +-- /commons-digester/commons-digester=2.1 <<< Conflict (direct=2.0)
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
[libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
[libversions] +-- /org.apache.hadoop/hadoop-common=2.2.0
[libversions]     +-- /com.sun.jersey/jersey-core=1.9 <<< Conflict (direct=1.8)
[libversions] ... and 15 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
[libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
[libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
[libversions]     +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0
[libversions]         +-- /com.sun.jersey/jersey-json=1.9
[libversions]             +-- /com.sun.xml.bind/jaxb-impl=2.2.3-1 <<< Conflict (direct=2.2.2)
[libversions] ... and 23 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework:
[libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3
[libversions] +-- /org.ow2.asm/asm=5.0_BETA <<< Conflict (direct=4.1)
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
[libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
[libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
[libversions]     +-- /log4j/log4j=1.2.17 <<< Conflict (direct=1.2.16)
[libversions] ... and 18 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) langid:
[libversions] /net.arnx/jsonic=1.2.7
[libversions] +-- /javax.servlet/servlet-api=3.0-alpha-1 <<< Conflict (direct=2.4)
[libversions] ... and 30 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) uima:
[libversions] /commons-digester/commons-digester=2.0
[libversions] +-- /commons-beanutils/commons-beanutils=1.8.0 <<< Conflict (direct=1.7.0)
[libversions] ... and 1 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
[libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
[libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
[libversions]     +-- /com.sun.jersey/jersey-server=1.9 <<< Conflict (direct=1.8)
[libversions] ... and 15 more
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) solrj:
[libversions] /org.apache.zookeeper/zookeeper=3.4.6
[libversions] +-- /io.netty/netty=3.7.0.Final <<< Conflict (direct=3.6.2.Final)
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework:
[libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3
[libversions] +-- /commons-io/commons-io=2.3 <<< Conflict (direct=2.1)
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework:
[libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3
[libversions] +-- /com.google.guava/guava=16.0.1 <<< Conflict (direct=14.0.1)
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core:
[libversions] /org.kitesdk/kite-morphlines-avro=0.12.1
[libversions] +-- /org.apache.avro/avro=1.7.5 <<< Conflict (direct=1.7.4)
[libversions] 
[libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core:
[libversions] /org.kitesdk/kite-morphlines-avro=0.12.1
[libversions] +-- /org.apache.avro/avro=1.7.5
[libversions]     +-- /org.xerial.snappy/snappy-java=1.0.5 <<< Conflict (direct=1.0.4.1)
[libversions] ... and 1 more
[libversions] Checked that ivy-versions.properties and ivy-ignore-conflicts.properties have lexically sorted '/org/name' keys and no duplicates or orphans.
[libversions] Scanned 44 ivy.xml files for rev="${/org/name}" format.
[libversions] Found 19 indirect dependency version conflicts.
[libversions] Completed in 20.55s., 12 error(s).
asfimport commented 10 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

I think this is ready to go. If there are no objections, I'll commit tomorrow.

asfimport commented 10 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

Thank you Steve! This is so useful.

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1598538 from @sarowe in branch 'dev/trunk' https://svn.apache.org/r1598538

LUCENE-5442: ant check-lib-versions will fail the build if there are unexpected version conflicts between direct and transitive dependencies.

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1598539 from @sarowe in branch 'dev/branches/branch_4x' https://svn.apache.org/r1598539

LUCENE-5442: ant check-lib-versions will fail the build if there are unexpected version conflicts between direct and transitive dependencies. (merged trunk r1598538)

asfimport commented 10 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Committed to trunk and branch_4x.

I'll open a follow-on issue to reduce the number of expected version conflicts listed in ivy-ignore-conflicts.properties, by upgrading the corresponding direct dependencies in ivy-versions.properties.

asfimport commented 10 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

I'll open a follow-on issue to reduce the number of expected version conflicts listed in {{ivy-ignore-conflicts.properties}}, by upgrading the corresponding direct dependencies in {{ivy-versions.properties}}.

Done: #6777

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1599134 from @sarowe in branch 'dev/trunk' https://svn.apache.org/r1599134

LUCENE-5442: smoke tester: 'ivy-ignore-conflicts.properties' is not a foreign invader

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1599138 from @sarowe in branch 'dev/branches/branch_4x' https://svn.apache.org/r1599138

LUCENE-5442: smoke tester: 'ivy-ignore-conflicts.properties' is not a foreign invader (merge trunk r1599134)