Use ivy cachepath and cachefileset instead of ivy retrieve [LUCENE-3943]

[asfimport]

In #5003 we moved to resolving all external dependencies using ivy:retrieve. This process places the dependencies into the lib/ folder of the respective modules which was ideal since it replicated the existing build process and limited the number of changes to be made to the build.

However it can lead to multiple jars for the same dependency in the lib folder when the dependency is upgraded, and just isn't the most efficient way to use Ivy.

Uwe pointed out that when working from svn or in using src releases we can remove the ivy:retrieve calls and make use of ivy:cachepath and ivy:cachefileset to build our classpaths and packages respectively, which will go some way to addressing these limitations – however we still need the build system capable of putting the actual jars into specific lib folders when assembling the binary artifacts

---

Migrated from LUCENE-3943 by Chris Male, updated Apr 03 2012

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Would this also mean ivy doesn't have to copy the JARs from its cache, into each checkout? So this will save disk space for devs w/ multiple checkouts...

[asfimport]

Chris Male (migrated from JIRA)

Correct

[asfimport]

Chris Male (migrated from JIRA)

We do this already for the maven-tasks dependency for example.

[asfimport]

Dawid Weiss (@dweiss) (migrated from JIRA)

This will require moving license checks till after the distribution is assembled, but it's a good idea. It's much like with Maven when things get stored once and IDEs and the build system reuses the same artifacts.

[asfimport]

Chris Male (migrated from JIRA)

That works with your goal of making it a top-level target, right?

[asfimport]

Dawid Weiss (@dweiss) (migrated from JIRA)

Yep, sure.

[asfimport]

Jan Høydahl (@janhoy) (migrated from JIRA)

+1. Perhaps with an ant target to explicitly list out all JARs currently on the classpath, for debugging?

[asfimport]

Chris Male (migrated from JIRA)

You can already see that with ant -v.

[asfimport]

Jan Høydahl (@janhoy) (migrated from JIRA)

-v gives you a lot but not a clear and concise listing of the classpath, including solr-jars and dependency jars?

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

updating summary to note that this will really only help when working with svn checkouts and src releases – we'll still need to actually copy jar files around when building the binary artifacts (which means we'll probably also still want something like clean-jar for use by the R.M.)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

we'll still need to actually copy jar files around when building the binary artifacts (which means we'll probably also still want something like clean-jar for use by the R.M.)

Is this really the case?

In my opinion, the ideal situation would be that we pass these filesets directly to the zip/tar/gz whatever in the binary release targets: they never see lib/, they are just added directly to the zip. So we could nuke the clean-jars stuff totally.

This is also way more ideal because then compile/test/release packaging tasks only change files in build/ and dist/, not elsewhere (be it tests modifying the source tree: SOLR-3268, or dist-maven modifying the source tree: #5017, or ivy:retrieve modifying the source tree: lib/), this is bad news and makes releasing complicated.

The tradeoff is we have to seriously modify packaging tasks to implement this, which I think is too risky to do for 3.x, but this would be a very nice improvement for making the release process less error-prone for 4.0 or possibly even 4.1 or later.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

In my opinion, the ideal situation would be that we pass these filesets directly to the zip/tar/gz whatever in the binary release targets

Ah... ok didn't think of that... +1

[asfimport]

Dawid Weiss (@dweiss) (migrated from JIRA)

In my opinion, the ideal situation would be that we pass these filesets directly to the zip/tar/gz whatever in the binary release targets

+1.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

In my opinion, the ideal situation would be that we pass these filesets directly to the zip/tar/gz whatever in the binary release targets

the one catch that occurs to me is the solr example: start.jar, the libraries jetty looks for, and the optional jar's solr load by path based on it's configuration ... we just have to make sure "ant example" takes care of putting all those jars where they need to be

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

ant example can also use ivy:cachepath as classpath, no need to copy anything.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

"ant example" doesn't run the example, so it can't use any ephemeral classpaths that ant creates on the fly. "ant example" currently sets up the example files (ie: copying the war to where jetty will look for it) hence my point that it could copy the jars as needed in order for jetty & solr to find them

(the example has to work in the binary build via "java -jar start.jar" even if users don't have any of the original build.xml files)