Replace Maven POM templates with full POMs, and change documentation accordingly [LUCENE-2657]

[asfimport]

The current Maven POM templates only contain dependency information, the bare bones necessary for uploading artifacts to the Maven repository.

The full Maven POMs in the attached patch include the information necessary to run a multi-module Maven build, in addition to serving the same purpose as the current POM templates.

Several dependencies are not available through public maven repositories. A profile in the top-level POM can be activated to install these dependencies from the various lib/ directories into your local repository. From the top-level directory:

mvn -N -Pbootstrap install

Once these non-Maven dependencies have been installed, to run all Lucene/Solr tests via Maven's surefire plugin, and populate your local repository with all artifacts, from the top level directory, run:

mvn install

When one Lucene/Solr module depends on another, the dependency is declared on the artifact(s) produced by the other module and deposited in your local repository, rather than on the other module's un-jarred compiler output in the build/ directory, so you must run mvn install on the other module before its changes are visible to the module that depends on it.

To create all the artifacts without running tests:

mvn -DskipTests install

I almost always include the clean phase when I do a build, e.g.:

mvn -DskipTests clean install

---

Migrated from LUCENE-2657 by Steven Rowe (@sarowe), 6 votes, resolved Jan 22 2011 Attachments: LUCENE-2657.patch (versions: 16), LUCENE-2657-branch_3x.patch (versions: 2) Linked issues:

• SOLR-1218
  • SOLR-1964

  • 2010

  • 1697

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I am amazed at how idea/eclipse ant tasks managed to get into trunk.

I already stated on the "maven thread", that I have no problem with something like this patch being committed to dev-tools/ alongside the eclipse and idea support... some others might object, but personally I see no problem.

we already agreed these folders are simply developer tools, and if they start to rot, we will just delete them.

my problem is with the release process and artifacts... so if we do this we need to be very clear that we are agreeing to put the maven support in dev-tools/ as "developer-support", but its not a build system, and it has no effect on artifacts... we don't release dev-tools, its just checked into svn to help developers.

[asfimport]

Chris Male (migrated from JIRA)

That was basically what I was getting at (perhaps not clearly enough).

Would a satisfactory compromise be to view this patch as adding development support for maven, which is not to do with whether maven artifacts are released or not?

The discussion about release process, artifacts and build system flamewars can then happen outside of this.

[asfimport]

Earwin Burrfoot (migrated from JIRA)

we need to be very clear .... and it has no effect on artifacts

I feel something was missed in the heat of debate. Eg:

The latest patch on this release uses the Ant artifacts directly.

bq. This patch uses the Ant-produced artifacts to prepare for Maven artifact publishing.

Maven itself is not invoked in the process. An Ant plugin handles the artifact deployment.

I will now try to decipher these quotes. It seems the patch takes the artifacts produced by Ant, as a part of our usual (and only) build process, and shoves it down Maven repository's throat along with a bunch of pom-descriptors. Nothing else is happening.

Also, after everything that has been said, I think nobody in his right mind will force anyone to actually use the Ant target in question as a part of release. But it's nice to have it around, in case some user-friendly commiter would like to push (I'd like to reiterate - ant generated) artifacts into Maven.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Chris: well thats the problem with maven, it tries to be too many things, a dependency management tool, a packaging system, a build system, ...

So, thats why I said we have to just be very clear about which exact scope of maven we are discussing. If the patch presented here is against /dev-tools, and is to assist developers who like maven, then as I said before I am totally ok with this, but I'm only speaking for myself.

Because maven is so many things, and due to Earwin's confusion, I think it would be good in general to add a README.txt to dev-tools anyway, that states what exactly it is (tools to assist lucene/solr developers, that aren't supported, its not bugs if they stop working, and will be deleted if they rot).

Separately what you said about other code in trunk is totally true... for example its my opinion that there is a lot of code in lucene's contrib that should be moved out to something like apache-extras... currently lucene's contrib has to compile and pass tests or the build fails... there is definitely some stuff in there that is more "sandboxy", slows down lucene core development, but itself isnt getting much maintenance other than devs doing the minimum work to make them pass tests... and we should be keep other options in mind for stuff like this.

[asfimport]

Earwin Burrfoot (migrated from JIRA)

Thanks, but I'm not the one confused here. : )

[asfimport]

Chris Male (migrated from JIRA)

Ant does many things too and we use it in a specific way so I see no problem defining what we intend our maven support to be for.

So I'm feeling some consensus (fortunately I spoke too soon before) that if we target this toward being a development tool which is not forced upon any users / release managers.

Is this okay with you Steven?

A README.txt describing the scope of the dev-tools sounds appropriate irrespective of what happens here. I certainly wasn't aware of what their maintenance plan was.

[asfimport]

Ryan McKinley (@ryantxu) (migrated from JIRA)

Steve, great work with this patch – it takes care of all the previous concerns about our problematic maven support.

With this patch, we now have:

• testable maven artifacts
• easy repo distribution
• ant is still the build system

The RM can choose to ignore the generate-maven-artifacts target and let someone else push the artifacts.

As with most religious conflicts – I hope the resolution is not conversion, rather something that lets everyone to live (work) in peace.

[asfimport]

Chris A. Mattmann (migrated from JIRA)

+1 for Steve's patch, great work and you beat me to it.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Final version of trunk patch. Added CHANGES.txt entries; converted old solr-tika-parsers-<version> dependency to tika-parsers-0.8; and changed Solr snapshot artifact generation to use default uniqueVersion value (true), like Lucene's.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

branch_3x patch. Maven build works, all tests pass, ant generate-maven-artifacts works.

There is one extra private artifact required in order to build with Java 5 (trunk required Java 6, so doesn't have this issue): solr/contrib/clustering/lib/carrot2-core-3.4.2.jar. The artifact in Maven central repo was compiled with Java 6, but the private artifact was specially compiled with Java 5. I added populating the local repo with this .jar to the bootstrap profile, and pulled the POM from the artifact in Maven central, to enable transitive dependency resolution.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

I plan on committing these patches tomorrow.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

as I said before... i really think the patches need to be against /dev-tools

[asfimport]

Ryan McKinley (@ryantxu) (migrated from JIRA)

i really think the patches need to be against /dev-tools

Sadly, given the way maven works, that is not possible. (without reinventing a crazy wheel)

Previous attempts at fakey fakey pom generation have not been so good – this approach is better since it is testable and works well with core maven tools.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Sorry, but introducing a parallel build system (somehow such that maven can "sneak" its way into being a 2nd supported build) is not acceptable to me. I feel this will slow down development. Putting it in /dev-tools clarifies the level of support: i don't want someone telling me "I broke the build" because these poms dont work if i refactor something.

So I am -1 to the patch, unless it is in /dev-tools. if maven cannot work with its files in /dev-tools, perhaps a patch to maven itself is needed instead.

[asfimport]

David Smiley (@dsmiley) (migrated from JIRA)

Perhaps a compromise would be to put them in /dev-tools and have an ant task that will copy (or symbolically link them when possible) to where Maven wants them? Developers opposed to maven stuff can choose to never invoke this ant task and thus leave the maven stuff hidden.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

David: sounds perfect to me, great idea.

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

Here's my suggestion: Let's put the stuff in and do a release. I am pretty sure that we have enough PMC members who want Maven that they can vote on said artifacts w/o those who don't want Maven to have to bother (even though they are the same artifacts). Also, Robert, I would specifically ask you, once Steve has done the work, assuming you are the RM, to honestly evaluate whether the production of the artifacts is a burden for you. If you don't want to, then I would suggest you do all the main RM work and you let one of us handle producing the artifacts to be released/voted on at the same time. If it really is a burden, then we can move it to dev-tools or external.

And, for the record, there are more than two camps for Maven besides "die maven die" and "maven or die": I personally think distributing artifacts via the central repository and the explicit requirement to list out dependencies and their versions is a great service and significantly increases uptake of artifacts, but I can live just fine without the day to day crap that is the result of running "mvn XXXX" (where XXXX is not equal to one of the simpler things like compile or test). Let's not be confused about what we are doing when we say we are supporting Maven. We are just using it to publish artifacts. The only time that should ever need to be run is during a release (hence it doesn't slow down us deciding to do a release, it just slows down, marginally, the actual release process.)

I totally agree we don't want two build systems and I am totally against switching to Maven as a build for all the reasons Robert is (parallel tests, etc.). All I have ever wanted is reliable published artifacts and I think we have enough support amongst committers to do that and we have strong enough interest in the community to warrant it.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

If it really is a burden, then we can move it to dev-tools or external.

Please re-read my comment above.

I won't stand for two build systems to deal with during development. If all you care about is reliable published artifacts, then its not an issue if the stuff to generate that is in /dev-tools or mixed in with the code.

I feel this is the maven users trying to sneak maven in as a fully supported build system, if not to then sneak it in as the only build system.

I will do everything possible to stop this.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

I'm sensitive to the two-build-systems problem. While I like being able to produce local Maven artifacts, I agree that mixed build systems for new users will be confusing.

I'm okay with putting the POMs themselves under dev-tools/. But I want the generate-maven-artifacts target to stay in Ant's build, not under dev-tools/, and a new target get-maven-poms added to the top-level build.xml; generate-maven-artifacts would depend on get-maven-poms. All pom.xml's would be added to svn:ignore entries where they are placed by get-maven-poms.

Robert, would you agree to keeping generate-maven-artifacts and adding get-maven-poms?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Steven, i totally dont mind if we change the whole way the ant build works with regards to maven, to depend on the dev-tools scenario.

For example, you could have 'ant maven' that:

1. copies the stuff from /dev-tools wherever it needs to be to make maven happy.
2. does whatever is necessary (invoke maven, whatever) to make the artifacts.

Or you could have 'ant maven' that only does part #1, but 'ant maven-artifacts' that does part #2, and depends on part #1.

i just want it to be in /dev-tools, because if i see build.xml in a directory, i think its supported by Ant. If i see Makefile in a directory, I think its supported by Make, pom.xml by Maven, and so on.

we could also add pom.xml to the svn:ignore for all the directories too, so that there are no problems with people using the maven framework in dev-tools.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

A chat on IRC confirms that Robert is okay with my proposal:

1. All POMs will be stored under dev-tools/maven/.
2. generate-maven-artifacts will stay in the Ant build.
3. A new get-maven-poms target will be added to the Ant build to copy the POMs from dev-tools/ to their target locations: in the top-level directory, and in each module's subdirectory under lucene/, solr/, and modules/ (trunk only).
4. svn:ignore properties will be changed to include pom.xml in their target locations.
5. generate-maven-artifacts will depend on get-maven-poms.

I'll work up modified patches for trunk and branch_3x along these lines.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Patches implementing my proposal to place the Maven POMs in dev-tools/maven/ and add a new top-level Ant target get-maven-poms, which is invoked by generate-maven-artifacts. generate-maven-artifacts remains in the top-level build.xml, as well as in lucene/, solr/, and modules/ (trunk only).

I couldn't figure out a way for generate-maven-artifacts under child directories lucene/, solr/ and modules/ to depend on the top-level get-maven-poms target, so I instead have generate-maven-artifacts in the child directories explicitly run the get-maven-poms target via the <ant> task. As a result, running generate-maven-artifacts from the top level will cause get-maven-poms to be run once for each child directory, but the repeated copy operation doesn't hurt anything, and the process is quick.

Unless there are objections, I will commit this tomorrow.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

+1, patch looks good.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Committed to trunk rev. 1061613, branch_3x rev. 1061612

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Uwe pointed out on IRC that the POMs under dev-tools/maven/ have a fixed version, and as it now is written, ant generate-maven-artifacts no longer respects the maven version property given on the command line.

This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts.

ant get-maven-poms will be responsible for the substitution and renaming, in addition to the copying from dev-tools/maven/ that it currently does.

[asfimport]

Chris Male (migrated from JIRA)

This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts.

Wasn't the fact we were using templates part of the problem that spawned this issue? I feel we've gone full circle here and are now back with maven templates that may or may not be maintained.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts.

Wasn't the fact we were using templates part of the problem that spawned this issue? I feel we've gone full circle here and are now back with maven templates that may or may not be maintained.

Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/, that will directly affect the quality/regularity of their maintenance.

In fact, I think whether the POMs are stored as templates under dev-tools/maven/ is really a non-issue, since they won't be used there. The way it's set up right now, if you want to use the POMs to drive a build, is that you run ant get-maven-poms to copy them over to their target locations. The way it will be set up once I convert the POMs under dev-tools/maven/ to templates is that you run ant get-maven-poms to copy them over to their target locations, as well as naming them to pom.xml and interpolating the desired version. In other words, for the end-user, the experience will be exactly the same: you run ant get-maven-poms, and then you can use Maven to build/test/install artifacts.

Templating the POMs will not change usability of them at all, so neither the number of people using them nor their effective maintenance will be affected by this change.

All that said, I agree with you that maintenance is the big deal here, and I welcome other suggestions about how to simplify and enable that process.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Rather than post a patch attachment for this change, I'll put the diffs here. I'll commit these shortly.

First, all pom.xml files under dev-tools/maven/ will be renamed to pom.xml.template.

Second, all pom.xml.template files will get this change (a representative version of it is given here - different original version for trunk's build.xml of course):

-  <version>3.1-SNAPSHOT</version>
+  <version>@version@</version>

Third, the top-level build.xml gets this change (different version for trunk's build.xml of course):

`@@` -39,10 +39,15 `@@`
     </sequential>
   </target>

+  <property name="version" value="3.1-SNAPSHOT"/>
   <target name="get-maven-poms" 
           description="Copy Maven POMs from dev-tools/maven/ to their target locations">
     <copy todir=".">
       <fileset dir="${basedir}/dev-tools/maven"/>
+      <filterset begintoken="@" endtoken="@">
+        <filter token="version" value="${version}"/>
+      </filterset>
+      <globmapper from="*.template" to="*"/>
     </copy>
   </target>

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Committed the template interpolation and renaming patches from my previous comment here, to trunk rev. 1062111 and branch_3x rev. r1062112

[asfimport]

David Smiley (@dsmiley) (migrated from JIRA)

Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/, that will directly affect the quality/regularity of their maintenance.

FWIW, I'll start using the generated POMs for my builds (aside from my use of IntelliJ for day to day development). If there are ever problems then I'll of course report/fix them. Thanks so much Steven for creating this alternative build system (one that I prefer).

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/, that will directly affect the quality/regularity of their maintenance.

FWIW, I'll start using the generated POMs for my builds (aside from my use of IntelliJ for day to day development). If there are ever problems then I'll of course report/fix them. Thanks so much Steven for creating this alternative build system (one that I prefer).

Cool! The more the merrier.

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

Bulk close for 3.1