apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.59k stars 1.01k forks source link

Replace Maven POM templates with full POMs, and change documentation accordingly [LUCENE-2657] #3731

Closed asfimport closed 13 years ago

asfimport commented 13 years ago

The current Maven POM templates only contain dependency information, the bare bones necessary for uploading artifacts to the Maven repository.

The full Maven POMs in the attached patch include the information necessary to run a multi-module Maven build, in addition to serving the same purpose as the current POM templates.

Several dependencies are not available through public maven repositories. A profile in the top-level POM can be activated to install these dependencies from the various lib/ directories into your local repository. From the top-level directory:

mvn -N -Pbootstrap install

Once these non-Maven dependencies have been installed, to run all Lucene/Solr tests via Maven's surefire plugin, and populate your local repository with all artifacts, from the top level directory, run:

mvn install

When one Lucene/Solr module depends on another, the dependency is declared on the artifact(s) produced by the other module and deposited in your local repository, rather than on the other module's un-jarred compiler output in the build/ directory, so you must run mvn install on the other module before its changes are visible to the module that depends on it.

To create all the artifacts without running tests:

mvn -DskipTests install

I almost always include the clean phase when I do a build, e.g.:

mvn -DskipTests clean install

Migrated from LUCENE-2657 by Steven Rowe (@sarowe), 6 votes, resolved Jan 22 2011 Attachments: LUCENE-2657.patch (versions: 16), LUCENE-2657-branch_3x.patch (versions: 2) Linked issues:

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

How would the BuildListener interface know about dependencies? Does it have some magic way to know this?

As an example, lets take modules/analysis/icu which has 3 dependencies:

take a look at modules/analysis/icu's pom.xml which has:

  <dependencies>
    <dependency>
      <groupId>com.ibm.icu</groupId>
      <artifactId>icu4j</artifactId>
      <version>${icu-version}</version>
    </dependency>
  </dependencies>

However, our ant builds (that depend on common-build/contrib-build) declare their dependencies in a semi-standard way:

The contrib-build.xml already has a 'dist-maven' target, that is called recursively. Perhaps an alternative would be to improve contrib-build.xml for it to have a 'generate-maven' target, also called recursively. I've already prototyped/proposed in SOLR-2002 that we migrate the solr build to extend the lucene build, so everywhere would use it.

Furthermore, couldnt we also make a recursive 'test-maven' target, that generates a maven project to 'download' or whatever it needs, then tries to run all the tests? If somehow the maven is broken, the tests simply won't pass.

I realize that running all of a modules tests again redundantly via 'maven' might not be the most elegant solution, but it seems like it would test that everything is working.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

How would the BuildListener interface know about dependencies? Does it have some magic way to know this?

BuildListener has hooks for build task onset and completion events (inter alia). ant2ide listens for Javac task completion, and captures from it the source and target directories, as well as the build classpath. You have to invoke compilation from an Ant build in order for this to work.

Seems kinda magical to me :)

The missing part here is figuring out the maven groupId/artifactId/version, and I think this can be dealt with by looking at the manifest in the jar. Maven-produced jars also contain their POMs, and pulling from there would be even simpler.

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

BuildListener has hooks for build task onset and completion events (inter alia). ant2ide listens for Javac task completion, and captures from it the source and target directories, as well as the build classpath. You have to invoke compilation from an Ant build in order for this to work.

Seems kinda magical to me

OK, i guess this is similar to when I open the project up in eclipse and say 'from existing ant build.xml file' and it sets up my eclipse project.

asfimport commented 13 years ago

Chris Male (migrated from JIRA)

I'm wondering whether generating poms is just going to make them more mysterious. While they are templates I can at least see them and they can be validated by XML validators (including IDEs). Suddenly they are created through a more complex process which we also have to maintain.

Assuming we have well-structured poms in the first place (huge assumption, different issue), are developers unprepared to add the appropriate dependency to the necessary pom? It doesn't seem like it'll happen very often, most of our artifacts have very few dependencies and we resist increasing them. We can document the process, make it very clear exactly what needs to be added and where.

Also, I'm not sure all jars are going to have the appropriate manifest. If the pom is generated (like Artifactory can do btw), I'm not sure it is packaged in the jar?

I think Robert was onto the right idea with this original proposal: If we consider our modules section and if we add a pom at the modules level and then one for each module, running a build from the modules level will recursively build each module. If there is something wrong with the poms themselves, maven will fail. If a dependency is omitted meaning the code cannot be built, maven will fail. If any of these 2 issues occur, then the build would fail and an issue can be created for those more comfortable with maven to fix.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Assuming we have well-structured poms in the first place (huge assumption, different issue), are developers unprepared to add the appropriate dependency to the necessary pom?

If nothing else, it is abundantly clear that there are more than a couple of Lucene/Solr developers that don't want to put in any effort for Maven's sake.

Also, I'm not sure all jars are going to have the appropriate manifest. If the pom is generated (like Artifactory can do btw), I'm not sure it is packaged in the jar?

I've looked into 10-15 jars, and very few have the required info. Some even have the wrong information in the manifest...

I think we can use .jar digest (SHA-1) as identifiers to look up the appropriate Maven info to automatically set up POM dependencies. I wrote a script to crawl the central Maven repo for SHA-1 keyed Maven coordinates (groupId:artifactId:version), and it seems to work, but at the rate it's progressing, it will take 4 days to finish. Anyway, it seems like overkill to gather info on all of the \~250k artifacts in the Maven central repo, just to handle the \~80 .jar files in the Lucene/Solr source tree.

I found a free service that provides Maven central repo search, including by artifact digest: http://www.jarvana.com/jarvana/search - it uses Lucene to power the search :). I plan on using this as the basis for automatic Maven coordinate lookup for arbitrary .jars.

asfimport commented 13 years ago

Chris Male (migrated from JIRA)

If nothing else, it is abundantly clear that there are more than a couple of Lucene/Solr developers that don't want to put in any effort for Maven's sake.

If developers are willing to add the dependency jar and lines to the ant file but not add the 5 lines to the pom, then I'm not sure any of these attempts are really going to succeed. Sure many developers don't like maven, but I haven't seen any straight out refuse to use it yet (unless I misread the large email thread), they are just demanding ease of use.

My concern here is this process is going in the opposite direction by making it vastly more complicated. What your proposing sounds like a great app independent of Lucene - a way to convert from an ant build system to a maven one, but I'm unsure its something that should be part of a build process instead of a once off thing. Certainly when we are trying to simplify said build process.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

My concern here is this process is going in the opposite direction by making it vastly more complicated. What your proposing sounds like a great app independent of Lucene - a way to convert from an ant build system to a maven one, but I'm unsure its something that should be part of a build process instead of a once off thing. Certainly when we are trying to simplify said build process.

It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it?

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

I don't see this issue as providing a complete Ant->Maven conversion system. The poms I envision will look just like the POM templates after they are interpolated - they will provide Maven coordinates for artifacts and their dependencies, nothing more. And the process to produce them can be carried out as part of generating Maven artifacts. I don't think they will need to be checked in, but they certainly could be.

In other words, I don't plan on building a complete Maven build configuration here. Just the bare-bones POMs required to upload artifacts, like the manually synched system we have now. Only without the manual synching part.

Vive la simplification!

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it?

I am trying to simplify the build, with the prototype in SOLR-2002, whereas most things are in a single common-build and we invoke them recursively (things like rat, dist, test, javadocs).

Right now it is complicated mostly because there are actually many build setups that are essentially standalone, and then have been linked to depend on each other.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it?

I am trying to simplify the build, with the prototype in SOLR-2002, whereas most things are in a single common-build and we invoke them recursively (things like rat, dist, test, javadocs).

I stand corrected. Build simplification is certainly a goal I support; without the flexibility afforded by a coherent build system, issues like this one would definitely be more difficult to implement.

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I stand corrected. Build simplification is certainly a goal I support; without the flexibility afforded by a coherent build system, issues like this one would definitely be more difficult to implement.

Exactly, because if you were to write a patch for this issue today, you have to implement it in 4 or 5 different places at the moment (or at least employ dirty ant hacks)

The same problem prevents us from really improving our build in other ways: for example we should probably support findbugs/pmd integration and things like that, and it should be simple.

asfimport commented 13 years ago

Chris Male (migrated from JIRA)

I don't see this issue as providing a complete Ant->Maven conversion system. The poms I envision will look just like the POM templates after they are interpolated - they will provide Maven coordinates for artifacts and their dependencies, nothing more. And the process to produce them can be carried out as part of generating Maven artifacts. I don't think they will need to be checked in, but they certainly could be.

Its this sort of middle ground that I'm unsure about. We basically have maven support today but not really. Some people understand how it works, but I've certainly seen users who are dabbling in some development and are confused by our maven support. I cant think of another project that generates its poms (even ant itself has the poms well-formed inside its codebase) so suddenly our support has become more unusual.

What I'm hoping is that we can actually add quality support for maven in such a way that users wouldn't be surprised, developers like you and I can get the benefits we know maven has, and that also simplifies what Robert has to do.

If all we can do at the moment is bolt bits of maven onto ant, I'm unsure its worth while.

asfimport commented 13 years ago

Chris Male (migrated from JIRA)

I should add that I'm currently looking at how ant supports maven releases since it does have a contrib-like folder of poms, well formed.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

I have changed the issue summary and description to manual generation of authoritative POMs, rather than automatic synching of POM templates with Ant dependencies, which can be pursued later.

asfimport commented 13 years ago

Yonik Seeley (@yonik) (migrated from JIRA)

Full Maven POMs will include the information necessary to run a multi-module Maven build

That sort of sounds like a parallel build process (i.e. you would be able to build lucene/solr itself with maven). Is it? We've avoided that type of thing in the past.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Full Maven POMs will include the information necessary to run a multi-module Maven build

That sort of sounds like a parallel build process (i.e. you would be able to build lucene/solr itself with maven). Is it? We've avoided that type of thing in the past.

Yes, it constitutes a parallel build process, but the Ant build process would remain the official build. For example, the artifacts produced by the Maven build will not be exactly the same as those produced by the Ant build process, and so cannot be used as release artifacts.

It has been noted elsewhere that a Maven build has never been produced for Lucene. I hope in this issue to provide that, so that the discussion about whether or not to include it in the source tree has a concrete reference point.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

First cut at Maven POMs for Lucene and Solr.

The following script, which installs non-mavenized artifacts into the user's local Maven repository, must be run before the Maven build works:

 mvn install:install-file -DgroupId=com.ibm.icu -DartifactId=icu4j -Dversion=4.4 -Dpackaging=jar -Dfile=modules/analysis/icu/lib/icu4j-4_4.jar
 mvn install:install-file -DgroupId=xerces -DartifactId=xercesImpl -Dversion=2.9.1-patched-XERCESJ-1257 -Dpackaging=jar -Dfile=lucene/contrib/benchmark/lib/xerces-2.9.1-patched-XERCESJ-1257.jar
 mvn install:install-file -DgroupId=com.sleepycat -DartifactId=berkeleydb -Dversion=4.7.25 -Dpackaging=jar -Dfile=lucene/contrib/db/bdb/lib/db-4.7.25.jar
 mvn install:install-file -DgroupId=com.sleepycat -DartifactId=berkeleydb-je -Dversion=3.3.93 -Dpackaging=jar -Dfile=lucene/contrib/db/bdb-je/lib/je-3.3.93.jar
 mvn install:install-file -DgroupId=org.apache.solr -DartifactId=solr-commons-csv -Dversion=4.0-SNAPSHOT -Dpackaging=jar -Dfile=solr/lib/commons-csv-1.0-SNAPSHOT-r966014.jar
 mvn install:install-file -DgroupId=org.apache.solr -DartifactId=solr-noggit -Dversion=r944541 -Dpackaging=jar -Dfile=solr/lib/apache-solr-noggit-r944541.jar
 mvn install:install-file -DgroupId=org.apache.tika -DartifactId=tika-core -Dversion=0.8-SNAPSHOT -Dpackaging=jar -Dfile=solr/contrib/extraction/lib/tika-core-0.8-SNAPSHOT.jar
 jar xvf solr/contrib/extraction/lib/tika-parsers-0.8-SNAPSHOT.jar META-INF/maven/org.apache.tika/tika-parsers/pom.xml
 mvn install:install-file -Dfile=solr/contrib/extraction/lib/tika-parsers-0.8-SNAPSHOT.jar -DpomFile=META-INF/maven/org.apache.tika/tika-parsers/pom.xml
 rm -rf META-INF

Everything compiles (using mvn -DskipTests clean install). Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF().

I have added configurations for running appassembler-maven-plugin (to produce run-time windows and linux scripts with lib/ dirs containing dependencies) for all classes with main() functions that are intended as tools rather than static tests.

I haven't tried using the POMs as project configuration for IntelliJ, so I'm not sure how that will go.

This is a work-in-progress. Please test and let me know if you find problems, or would like to see more stuff added, or if you think the design is all wrong.

One challenge worth noting is that Maven disallows aggregator POMs (those with a <modules> section directing nested Maven invocations) from producing any artifacts other than the POM itself. As a result, the 4 or 5 locations in the Lucene/Solr build that have "mixed content" (to borrow the XML term for mixed nested element and text content) have to be fudged; e.g. the Lucene and Solr core POMs are located in lucene/src/ and solr/src/, because the aggregator POMs at lucene/ and solr/ can't produce binary artifacts. (The conventional Maven way to address this kind of issue is to make the "core" modules be siblings of the nested modules.)

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF().

maybe it is an icu classpath issue. modules/analysis and solr/contrib/extraction both use different, out of date versions. i opened #3757

asfimport commented 13 years ago

Chris Male (migrated from JIRA)

Hi Steven,

The following script, which installs non-mavenized artifacts into the user's local Maven repository

Would system scope dependencies be an alternative way? Then we can specify the systemPath to where the jars are in the lib folders. Reason I suggest this is that if someone replaces one of those jar (which will happen I'm sure) or simply adds another, then we cannot simply update the appropriate pom. We'd also need to install the jars locally (possibly again).

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Would system scope dependencies be an alternative way?

I thought of that, and it probably makes sense for most of them.

But a jar like tika-parsers-0.8-SNAPSHOT.jar, with 10 or 15 dependencies, would be even messier to deal with as a system dependency than as a locally installed artifact. We would have to go look at the POM for it (contained conveniently within the jar, as it happens), compare the dependencies against the versions of the local dependencies (which would have previously had to be transferred to the both the Solr Cell and the DIH Extras POMs) and make changes in both places.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

This version of the patch puts the third-party non-mavenized .jar installation stuff into a profile in the top-level POM.

So instead of running the above-listed script, instead just run the following Maven invocation:

mvn -N -Pbootstrap install

-N disables recursive building, -Pbootstrap activates the "bootstrap" profile (defined in the POM) and the local repo artifact installation stuff is bound to the "install" phase in the "bootstrap" profile.

I discovered that the Tika artifact snapshots (tika-core and tika-parsers) are hosted at the Apache snapshots repository, and that was causing these dependencies not to work properly, so I switched their coordinates to org.apache.solr:solr-tika-*:4.0-SNAPSHOT, so that they will remain stable.

Also, the previous patch had some weirdness with the Solr DIH Extras POM - it wasn't being treated as a new file, but rather a diff from some other file (the POM template, I think) - anyway, this patch fixes the problem.

Everything still compiles, and all tests except the two previously failing tests still pass.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF().

maybe it is an icu classpath issue. modules/analysis and solr/contrib/extraction both use different, out of date versions. i opened #3757

I dropped in the ICU4J 4.4.2 jar, and everything compiled and all tests succeeded under Maven except for the same two tests, which continued to fail.

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I dropped in the ICU4J 4.4.2 jar, and everything compiled and all tests succeeded under Maven except for the same two tests, which continued to fail.

Well it wasn't the icu issue then, but we needed to upgrade anyway for the bugfixes, no loss.

What failure are you seeing? When prototyping the the ant system refactor (SOLR-2002 / SOLR-2002_merged.patch), i had a few issues with some tests but I don't remember the details. I remember it was really important to get the CWD of the junit run correct, e.g. in that patch its:

<property name="tests.userdir" location="src/test/resources"/>
  and later invoked as junit's dir="${tests.userdir}" attribute
asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

What failure are you seeing?

Two apparently unrelated errors:

  1. JettyWebappTest.testJSP() returns an HTTP 500 error as soon as it attempts to connect to the Jetty Server instantiated in setUp(), on line #88. I haven't figured out how to set Jetty Logging up so that I have visibility on what's happening yet. I see a call setting the solr.solr.home property in setUp() on line #49, so maybe that value is not in sync with the current directory I transferred from the Ant configuration.
  2. solr/contrib/extraction's ExtractingRequestHandlerTest.testArabicPDF() test fails because no result is found containing the extracted Arabic term (line #353). I have traced content extraction up through plain text extraction from the PDF, and that seems to be working, so I don't think it's a classpath issue with Tika or PDFBox. I need to keep tracing to see happens after the text extraction.
asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Both of the failing tests were caused by missing dependencies: jetty jsp-2.1 (scope=provided) was missing from solr/src/pom.xml, and icu4j was missing from solr/contrib/extraction/pom.xml.

With this version of the patch, all tests now pass.

IntelliJ IDEA project import from the Maven configuration mostly works, with the exception that because the Solr core and Solrj modules each include two source directories, and IntelliJ doesn't grok this fact from the POMs (specified via build-helper-maven-plugin configuration), builds including these two modules fail, since they aren't seeing all of their source code.

http://youtrack.jetbrains.net/issue/IDEA-27176 has a work-around for this issue: click on "update folders for all projects" button in the maven projects pane - this works for Solr core (webapp/src/ is added a source root), but not for Solrj, likely because the second source root solr/src/common/ is a sibling of the module root solr/src/solrj/, rather than a child.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

icu4j was missing from solr/contrib/extraction/pom.xml

See SOLR-1813 for details.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Synch'd to trunk (ICU4J and Xerces-J dependency upgrades). Use mvn -N -Pbootstrap install from the top-level directory in order to install the new dependencies.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Updated to trunk, including addition of a POM for solr/contrib/analysis-extras/ and upgraded dependencies.

All tests pass.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Earlier Maven work

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Changes in this version of the patch:

  1. Reduced configuration redundancy: as much as possible, shared configuration is now located in the lucene-parent POM.
  2. The solr-aggregator POM is converted to a solr-parent POM, and all Solr modules are now children of it rather than of lucene-parent. This POM hosts Solr-specific shared configuration.
  3. The solr war is now generated in a new POM located in solr/src/webapp/. This POM also includes an untested maven-jetty-plugin configuration, for use in starting up Solr from Maven, as requested in SOLR-1218.
  4. A profile in the lucene-parent POM to generate source jars for each module: source-jar.
  5. A profile in the lucene-parent POM to generate javadoc jars for each module: javadocs-jar.
  6. Consistent MANIFEST.MF entries in all archives, including timestamps generated using embedded Groovy via gmaven-plugin and the Subversion revision from buildhelper-maven-plugin.
  7. NOTICE.txt and LICENSE.txt are now placed in META-INF/ in all archives
  8. The Maven plugin versions are updated to the latest available.

All tests pass.

If you have used previous versions of the patch, you should re-run mvn -N -P bootstrap install from the top level, to install the non-maven dependencies in your local Maven repository. (Some of the jar names have changed to better reflect their names as used in Solr.)

To create binary, javadoc and source jars, and install them in your local repository, run the following at any level:

mvn -DskipTests -P javadocs-jar,source-jar clean install
asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

All tests pass again with this patch. Solr test resource structrual changes from SOLR-2299 (r1054169) are now accounted for. All test CWDs are set to ${project.build.testOutputDirectory}, so that any stray files put on the CWD by tests will be cleaned up using either mvn clean or ant clean.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Resync with trunk to move lucene/contrib/benchmark/ to modules/benchmark/: #3919

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Removed lucene/contrib/remote/ (#3911)

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Added profiles to populate internal repositories at lucene/dist/maven/ and solr/dist/maven/ with generated artifacts.

To populate lucene/dist/maven/ with POMs and binary, source and javadoc artifacts, run the following from the top level Lucene/Solr directory:

mvn -N -P bootstrap,deploy-to-lucene-dist-maven-repository deploy
cd lucene
mvn -DskipTests -Pdeploy-to-lucene-dist-maven-repository,javadocs-jar,source-jar deploy
cd ../modules
mvn -DskipTests -Pdeploy-to-lucene-dist-maven-repository,javadocs-jar,source-jar deploy

To populate solr/dist/maven/, run the following from the top level Lucene/Solr directory:

mvn -N -P bootstrap install
cd solr
mvn -DskipTests -Pdeploy-to-solr-dist-maven-repository,javadocs-jar,source-jar deploy
asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

carrot2 dependency upgraded to 3.4.2: SOLR-2296

asfimport commented 13 years ago

Ryan McKinley (@ryantxu) (migrated from JIRA)

This is looking good!

IIUC, this will be a parallel build system to ant. The build and test is independent of anything the ant build does.

If we take this route, we should probably drop the -pom.xml.templates and the and --generate-maven-artifacts target.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

IIUC, this will be a parallel build system to ant. The build and test is independent of anything the ant build does.

Yes, except that the two systems share build output directories.

If we take this route, we should probably drop the -pom.xml.templates and the and --generate-maven-artifacts target.

I agree.

The -pom.xml.templates have never been fully correct (e.g. missing dependencies) and are unmaintained.

I'm working on replacing generate-maven-artifacts.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

To populate both lucene/dist/maven/ and solr/dist/maven/, run from the top level:

mvn -Pdist -DskipTests deploy

To populate only lucene/dist/maven/, run from the top level:

mvn -N -Pdist deploy
cd lucene
mvn -Pdist -DskipTests deploy
cd ../modules
mvn -Pdist -DskipTests deploy
asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

This patch cuts over ant generate-maven-artifacts to directly use mvn. After applying this patch, Maven 2.2.1 (and maybe 3.0.X? - untested) must be installed on your machine in order to run ant generate-maven-artifacts.

Other changes in this patch:

  1. Dropped all use of Maven Ant Tasks.
  2. The top-level ant generate-maven-artifacts now works and is the best way to perform this task, since it will create a single timestamp for all artifacts; this target can also be run from either solr/ or lucene/.
  3. Removed the generate-maven-artifacts target from modules/build.xml, and transferred the responsibility for generating modules/* maven artifacts to lucene/build.xml.
  4. The solr/src/webapp/ module no longer installs or deploys its (empty) sources jar.
  5. Remote Maven artifact deployment is no longer included in the Ant build - this can be performed by the Maven build.
  6. mvn clean no longer removes solr/dist/ or lucene/dist/, for two reasons:
    1. The Ant build populates dist/ with things that the Maven build should not remove. Removing just dist/maven/ won't work, because:
    2. I couldn't find a nice/simple way to remove a directory just once in the reactor build. The previous patch attempted to do this from the lucene core and solr core modules, but that solution was deleting their deployed parent POMs, since the reactor build orders the Solr and Lucene parent POMs before the core modules (the parent relationship requires this).

Left to do:

  1. Add Ant targets to test the Maven artifacts
  2. Backport to branch_3x
asfimport commented 13 years ago

Chris A. Mattmann (migrated from JIRA)

Hey Guys,

I've set this up on some other Apache projects (Nutch, Tika [NetCDF4] and SIS so far), and basically it involved:

  1. moddin'g build.xml according to Sonatype's guide (see build.xml section)

https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

  1. adding pom.xmls for each artifact to be published

I'll throw together a patch for this and see if I can't make this process a bit easier.

Thanks.

Cheers, Chris

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

In this patch:

  1. ant generate-maven-artifacts now works the same as it does on trunk without this patch – using maven-ant-tasks – except that instead of using the POM templates, the POMs provided in the patch are used.
  2. ant generate-maven-artifacts now functions properly at the top level, from lucene/, from modules/, and from solr/.
  3. Removed all *-source.jar and *-javadoc.jar generation related functionality from the POMs, as well as the dist profile - the Ant build is responsible for putting together the maven artifacts.
  4. Removed the POM templates, except for the two required to deploy the solr-noggit and solr-commons-csv artifacts from the Ant build.
  5. Modified the Maven artifact handling in the Ant build, including artifact signing, to be correct.
  6. Based on feedback from Stevo Slavić <http://www.mail-archive.com/solr-user@lucene.apache.org/msg45656.html&gt;, added explicit <groupId>'s to the POMs that didn't have them, and added explicit <relativePath>'s to the <parent> declarations in all POMs.

I think this patch is ready to be committed to trunk.

I'll post a branch_3x version of this patch tomorrow, and then I think the patches on this issue will be complete.

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I think this patch is ready to be committed to trunk.

Well first of all, you obviously worked hard on this, but we need to think this one through before committing. Can we put this code in a separate project, that takes care of maven support for lucene? The problem is there are two camps "die maven die" and "maven or die". There will never be consensus.

The only way for maven to survive, is for the users that care about it, to support itself, just like other packaging systems such as debian, redhat rpm, freebsd/mac ports, etc etc that we lucene, don't deal with. They can't continue to whine to people like me, who don't give a shit about it, to support it and produce its crazy ass complicated artifacts.

Instead the people who care about these packaging systems, and know how to make them work must deal with them.

Personally I really don't like:

Basically, the ant build system is our build. I think it needs to be able to fully build lucene for a release without involving any other build systems such as Make or Maven.

asfimport commented 13 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Can we put this code in a separate project, that takes care of maven support for lucene?

I'd rather not. The Lucene project has published Maven artifacts since the 1.9.1 release. I think we should continue to do that.

The only way for maven to survive, is for the users that care about it, to support itself, just like other packaging systems such as debian, redhat rpm, freebsd/mac ports, etc etc that we lucene, don't deal with.

OK, those are pretty obviously red herrings. Can we concentrate on the actual issue here without dragging in those extraneous things? Maven artifacts, not those other things, have been provided by Lucene since the 1.9.1 release. We obviously do deal with Maven.

They can't continue to whine to people like me, who don't give a shit about it, to support it and produce its crazy ass complicated artifacts.

The latest patch on this release uses the Ant artifacts directly. POMs are provided. You know, just like it has been since the 1.9.1 release.

Instead the people who care about these packaging systems, and know how to make them work must deal with them.

Um, like the patch on this issue is doing?

Basically, the ant build system is our build. I think it needs to be able to fully build lucene for a release without involving any other build systems such as Make or Maven.

This patch uses the Ant-produced artifacts to prepare for Maven artifact publishing. Maven itself is not invoked in the process. An Ant plugin handles the artifact deployment.

I seriously do not understand why this is such a big deal. Why can't we just keep publishing Maven artifacts? You know, like we have for the past 15-20 releases.

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I'd rather not. The Lucene project has published Maven artifacts since the 1.9.1 release. I think we should continue to do that.

Well I disagree, and I definitely don't think we should have 2 build systems.

As I stated before, I don't think we are going to see consensus on this issue, because maven is too "imposing" on people who don't care about it... it seems maven users are only happy if every project they depend on is itself a maven project.

asfimport commented 13 years ago

Chris Male (migrated from JIRA)

I'm a little lost at what this patch introduces that is imposing? Ant itself has maven support as part of its trunk code base so its clearly not too imposing for them.

Is your issue that this patch introduces things that get in your way somehow with using ant to do builds? or are you against committing this due to your general concerns with Maven?

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Is your issue that this patch introduces things that get in your way somehow with using ant to do builds? or are you against committing this due to your general concerns with Maven?

Personally I don't care for maven, but I would be bringing up these same objections if we proposed building Redhat RPMs or FreeBSD ports. I would say, its best for us to stay out of the business of building packages for various systems, instead let the people that work with those packaging systems do it themselves, they will do a better job and supporting N build/packaging systems is too expensive for us. It makes releasing more complicated to deploy these special packages to the various repositories where they want to be, it forces the release manager to know about N packaging systems instead of concentrating on releasing the source code to an open source search engine library, which is what we should be doing.

asfimport commented 13 years ago

Chris Male (migrated from JIRA)

Alright I can appreciate your concern. I think comparing Maven to RPM or FreeBSD ports is going a little far, but I can understand the point you're making.

What if this were committed so that those of us who do understand maven and do like using it, could?
This issue about whether maven artifacts need to then be released or not can be part of a greater discussion (as is already taking place).

By committing this we then make it easier for someone else outside of the project to create the correct artifacts which are then available from the central maven repository, if thats the decision thats made which is also the one you support.

asfimport commented 13 years ago

Robert Muir (@rmuir) (migrated from JIRA)

What if this were committed so that those of us who do understand maven and do like using it, could? This issue about whether maven artifacts need to then be released or not can be part of a greater discussion (as is already taking place).

By committing this we then make it easier for someone else outside of the project to create the correct artifacts which are then available from the central maven repository, if thats the decision thats made which is also the one you support.

Why does it need to be committed to our trunk? why can't it be on e.g. apache-extras and do all of this.

This way, we can continue to evolve lucene (e.g. merge lucene and solr better), maintaining only ONE build system, and hopefully! release more often... if downstream maven support falls behind, it falls behind, but it shouldn't slow us down.

asfimport commented 13 years ago

Chris Male (migrated from JIRA)

I can appreciate the need to keep our trunk lean and mean and I hope that your assertion that that'll mean more frequent releases holds true. I hope we also shine the light on other code in the trunk to see if it belongs there or could be spun off.

I do continue to support the committing of this code but as you rightly pointed out, there won't be any consensus on the issue so I'll leave it at that.

asfimport commented 13 years ago

Earwin Burrfoot (migrated from JIRA)

<irony> I am amazed at how idea/eclipse ant tasks managed to get into trunk. No, really, emacs is just enough for Lucene development, as proven by one of our most seasoned developers. So why the hell are those extras polluting main tree? </irony>