search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.66k stars 1.03k forks source link

nuke jars from source tree and use ivy [LUCENE-3930] #5003

Closed asfimport closed 12 years ago

asfimport commented 12 years ago

As mentioned on the ML thread: "switch jars to ivy mechanism?".

Migrated from LUCENE-3930 by Robert Muir (@rmuir), resolved Apr 02 2012 Attachments: ant_-verbose_clean_test.out.txt, langdetect-1.1.jar, LUCENE-3930__ivy_bootstrap_target.patch, LUCENE-3930_includetestlibs_excludeexamplexml.patch, LUCENE-3930.patch (versions: 3), LUCENE-3930-skip-sources-javadoc.patch, LUCENE-3930-solr-example.patch (versions: 2), noggit-commons-csv.patch, patch-jetty-build.patch, pom.xml Linked issues:

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

prototype patch for the lucene core.

You dont need to do anything special for this patch to work: 'ant test' etc works as usual (though only for lucene-core!!!)

i nuked all the libs in lucene/lib/ (junit, ant, ant-junit).

These are instead dependencies of the test-framework.

I have no idea what uses this maven-ant-tasks.jar thats still there, or where to fetch it from, but maybe Steve knows :)

asfimport commented 12 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

I have no idea what uses this maven-ant-tasks.jar thats still there, or where to fetch it from, but maybe Steve knows :)

maven-ant-tasks.jar is used by generate-maven-artifacts / dist-maven / m2-deploy\* to deploy Maven artifacts; these are used by the Jenkins Maven builds to deploy snapshots to the Apache Snapshot repository, and up through the most recent release, to create Maven release artifacts to be published on Maven central.

I unpacked the .jar and found a POM under META-INF, and it says its groupId:artifactId:version coordinates are org.apache.maven:maven-ant-tasks:2.1.3, which is available from the Maven Central repository at http://repo1.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/.

asfimport commented 12 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

so you could simply fetch it with ivy and make it available to ant using <ivy:cachepatch pathid="foobar"/> and then load the targets from the classpath in classpathref="foobar".

Like that:

<ivy:cachepath organisation="emma" module="emma" revision="2.0.4217" inline="true" conf="ant" pathid="emma.classpath"/>
<taskdef resource="emma_ant.properties" classpathref="emma.classpath" />

Thanks!

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

so you could simply fetch it with ivy and make it available to ant using <ivy:cachepatch classpathref="foobar"/> and then load the targets from the classpath in ref="foobar".

I'm not gonna do that now: thats an optimization. Various ant tasks expect libs to be in lib/, so I want to populate lib/, not rewrite the build system.

asfimport commented 12 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

I already implemented it :-)

asfimport commented 12 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Here the patch with the maven-ant-tasks. It works theoretcally, but the dependent tasks dont completely work at the moment.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Thanks: i will look at maven-ant-tasks. I have dependent tasks working (e.g. all of lucene+contrib+modules) currently... patch coming soon.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

updated patch:

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I will make a heavy committing branch for this issue. I think we can solve this soon!

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Created branch: https://svn.apache.org/repos/asf/lucene/dev/branches/lucene3930

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Great stuff Robert. I've come into the discussion a little late but I'll help where I can.

bogusly every contrib/module currently needs an ivy.xml (even with no dependencies). I'd like to remove this but i just havent mucked with it yet.

I don't think this is so bad? It shows that the contrib/module is properly configured and that the ivy integration was 'done' when the module was created. I think it might also help IDEs if every module is consistent.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Committed Uwe's patch for dist-maven. I also added OOM Protection (©), but its still broken for the modules/analysis which recurses into all the submodules. This needs to be fixed to work like contrib-crawl/modules top-level crawl. But this is nothing new and really unrelated to this issue...

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Should I move the notice files from lucene/lib to lucene/test-framework/lib?

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

they should be moved. also test-framework/lib should get an svn:ignore of *.jar

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Patch for addressing Solr's example/lib jars. Adds a build.xml and ivy.xml to solr/example

I haven't committed this since I'm unsure its the best way forward. I've explored other options with ivy (through the use of confs for example) and there doesn't seem to be any other clean way.

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

This has a few other benefits. We can move the other targets related to the example to example/build.xml and tie in the resolve target.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Lets defer any unnecessary refactoring (even if it has benefits) to other issues.

I think we want the minimal (reasonable) patch here that works as close as possible to the ant build: given the incubator discussion about jars, unfortunately I think we need to backport this to 3.6.

So its important to keep everything as close as possible to what it was before so that packaging tasks, etc work.

Seeing how things got working for solr I think we may want to revert even the test-framework/lib and go back to lucene/lib intentionally just for this purpose: I'll look into this.

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Lets defer any unnecessary refactoring (even if it has benefits) to other issues.

Absolutely, just adding more information to chew over here.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I think the patch is fine, but it doesn't need to override ivy.retrieve.pattern, example/lib is already its default.

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Patch without pattern override. I'll pick up where I left off in the AM.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Thanks for all the help!!!!!!!

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Now we have the problematic patched jars and the renamed unreleased jars.

I will start with the patched ones: these are the worst case. I will look to see if we can have our patch file sitting in svn, we download the sources (not jar), patch with the patch file, and compile the thing up.

asfimport commented 12 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

FWIW.... i did a completley clean checkout of the lucene3930 (r1306662) and got the following build failure trying to run "ant clean test" from the top level.

the ivy bootstraping doesn't seem to play nicely with the license checker...

hossman@bester:\~/lucene/branch_lucene3930$ ant clean test
Buildfile: build.xml

clean:

clean:

clean:

clean:
     [echo] Building analyzers-common...

clean:
     [echo] Building analyzers-icu...

clean:
     [echo] Building analyzers-kuromoji...

clean:
     [echo] Building analyzers-morfologik...

clean:
     [echo] Building analyzers-phonetic...

clean:
     [echo] Building analyzers-smartcn...

clean:
     [echo] Building analyzers-stempel...

clean:
     [echo] Building analyzers-uima...

clean:
     [echo] Building benchmark...

clean:
     [echo] Building facet...

clean:
     [echo] Building grouping...

clean:
     [echo] Building join...

clean:
     [echo] Building queries...

clean:
     [echo] Building queryparser...

clean:
     [echo] Building spatial...

clean:
     [echo] Building suggest...

clean:
     [echo] Building solr...

clean:

validate:

compile-tools:

download-ivy:
    [mkdir] Created dir: /home/hossman/lucene/branch_lucene3930/lucene/ivy
     [echo] 
     [echo]       NOTE: You do not have ivy installed, so downloading it...
     [echo]       If you make lots of checkouts, download ivy-2.2.0.jar yourself 
     [echo]       and set ivy.jar.file to this in your \~/build.properties
     [echo]       
      [get] Getting: http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar
      [get] To: /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar

install-ivy:

resolve:
[ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: url = jar:file:/home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml

init:

compile-core:
    [mkdir] Created dir: /home/hossman/lucene/branch_lucene3930/lucene/build/tools/classes/java
    [javac] Compiling 2 source files to /home/hossman/lucene/branch_lucene3930/lucene/build/tools/classes/java
     [copy] Copying 1 file to /home/hossman/lucene/branch_lucene3930/lucene/build/tools/classes/java

validate:
     [echo] License check under: /home/hossman/lucene/branch_lucene3930/lucene
 [licenses] MISSING LICENSE for the following file:
 [licenses]   /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar
 [licenses]   Expected locations below:
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-ASL.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-BSD.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-BSD_LIKE.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-CDDL.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-CPL.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-EPL.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-MIT.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-MPL.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-PD.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-SUN.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-COMPOUND.txt
 [licenses]   => /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-FAKE.txt
 [licenses] Scanned 1 JAR file(s) for licenses (in 0.31s.), 1 error(s).

BUILD FAILED
/home/hossman/lucene/branch_lucene3930/build.xml:42: The following error occurred while executing this line:
/home/hossman/lucene/branch_lucene3930/lucene/build.xml:178: The following error occurred while executing this line:
/home/hossman/lucene/branch_lucene3930/lucene/tools/custom-tasks.xml:22: License check failed. Check the logs.

Total time: 5 seconds
asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Thanks hoss, I will add in a LICENSE and NOTICE for ivy.

asfimport commented 12 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

I did a touch /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-FAKE.txt to work around the license checker and this time i wound up with an OOM...

hossman@bester:\~/lucene/branch_lucene3930$ ant clean test
....
common.compile-test:
    [mkdir] Created dir: /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/sandbox/classes/test
    [javac] Compiling 6 source files to /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/sandbox/classes/test
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
     [copy] Copied 1 empty directory to 1 empty directory under /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/sandbox/classes/test

test-contrib:
     [echo] Building demo...

download-ivy:

install-ivy:

resolve:
[ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: url = jar:file:/home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
    at java.lang.Throwable.getStackTraceElement(Native Method)
    at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
    at java.lang.Throwable.printStackTrace(Throwable.java:462)
    at java.lang.Throwable.printStackTrace(Throwable.java:451)
    at org.apache.tools.ant.Main.startAnt(Main.java:230)
    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
    at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)

running with "-verbose" to see if i can get more details on exactly where/why the OOM is happening

asfimport commented 12 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

here's the tail end of "ant -verbose clean test"...

hossman@bester:\~/lucene/branch_lucene3930$ ant -verbose clean test

...

compile-test-framework:
Skipped because property 'lucene.test.framework.compiled' set.

common.compile-test:
    [mkdir] Skipping /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test because it already exists.
Property "run.clover" has not been set
    [javac] org/apache/lucene/demo/TestDemo.java omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/TestDemo.class is up to date.
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/apache1.0.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/apache1.1.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/apache2.0.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/cpl1.0.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/epl1.0.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/freebsd.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/gpl1.0.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/gpl2.0.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/gpl3.0.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/lgpl2.1.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/lgpl3.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/lpgl2.0.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/mit.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/mozilla1.1.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/mozilla_eula_firefox3.txt skipped - don't know how to handle it
    [javac] /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/src/test/org/apache/lucene/demo/test-files/docs/mozilla_eula_thunderbird2.txt skipped - don't know how to handle it
     [copy] org/apache/lucene/demo/test-files/docs/apache1.0.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/apache1.0.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/apache1.1.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/apache1.1.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/apache2.0.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/apache2.0.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/cpl1.0.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/cpl1.0.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/epl1.0.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/epl1.0.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/freebsd.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/freebsd.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/gpl1.0.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/gpl1.0.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/gpl2.0.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/gpl2.0.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/gpl3.0.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/gpl3.0.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/lgpl2.1.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/lgpl2.1.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/lgpl3.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/lgpl3.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/lpgl2.0.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/lpgl2.0.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/mit.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/mit.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/mozilla1.1.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/mozilla1.1.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/mozilla_eula_firefox3.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/mozilla_eula_firefox3.txt is up to date.
     [copy] org/apache/lucene/demo/test-files/docs/mozilla_eula_thunderbird2.txt omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs/mozilla_eula_thunderbird2.txt is up to date.
     [copy]  omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test is up to date.
     [copy] org omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org is up to date.
     [copy] org/apache omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache is up to date.
     [copy] org/apache/lucene omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene is up to date.
     [copy] org/apache/lucene/demo omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo is up to date.
     [copy] org/apache/lucene/demo/test-files omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files is up to date.
     [copy] org/apache/lucene/demo/test-files/docs omitted as /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/demo/classes/test/org/apache/lucene/demo/test-files/docs is up to date.
  [antcall] Exiting /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/build.xml.

compile-tools:
Detected Java version: 1.6 in: /usr/lib/jvm/java-6-sun-1.6.0.24/jre
Detected OS: Linux
Project base dir set to: /home/hossman/lucene/branch_lucene3930/lucene/tools
      [ant] calling target(s) [compile-core] in build file /home/hossman/lucene/branch_lucene3930/lucene/tools/build.xml
parsing buildfile /home/hossman/lucene/branch_lucene3930/lucene/tools/build.xml with URI = file:/home/hossman/lucene/branch_lucene3930/lucene/tools/build.xml
Project base dir set to: /home/hossman/lucene/branch_lucene3930/lucene/tools
Importing file /home/hossman/lucene/branch_lucene3930/lucene/common-build.xml from /home/hossman/lucene/branch_lucene3930/lucene/tools/build.xml
parsing buildfile /home/hossman/lucene/branch_lucene3930/lucene/common-build.xml with URI = file:/home/hossman/lucene/branch_lucene3930/lucene/common-build.xml
Already defined in main or a previous import, ignore compile-core
Already defined in main or a previous import, ignore javadocs
 [property] Loading /home/hossman/lucene.build.properties
 [property] Unable to find property file: /home/hossman/lucene.build.properties
 [property] Loading /home/hossman/build.properties
 [property] Unable to find property file: /home/hossman/build.properties
 [property] Loading /home/hossman/lucene/branch_lucene3930/lucene/tools/build.properties
 [property] Unable to find property file: /home/hossman/lucene/branch_lucene3930/lucene/tools/build.properties
 [property] Loading /home/hossman/lucene/branch_lucene3930/lucene/build.properties
 [property] Unable to find property file: /home/hossman/lucene/branch_lucene3930/lucene/build.properties
Override ignored for property "DSTAMP"
Override ignored for property "TSTAMP"
[available] Found: /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar
Property "run.clover" has not been set
[available] Found: javadoc/java6/package-list
Override ignored for property "build.dir"
[available] Unable to load class com.cenqua.clover.tasks.CloverReportTask to set property clover.present
Importing file /home/hossman/lucene/branch_lucene3930/lucene/tools/custom-tasks.xml from /home/hossman/lucene/branch_lucene3930/lucene/common-build.xml
parsing buildfile /home/hossman/lucene/branch_lucene3930/lucene/tools/custom-tasks.xml with URI = file:/home/hossman/lucene/branch_lucene3930/lucene/tools/custom-tasks.xml
 [macrodef] creating macro  license-check-macro
Trying to override old definition of task m2-deploy
 [macrodef] creating macro  m2-deploy
 [macrodef] creating macro  m2-deploy-with-pom-template
Trying to override old definition of task build-manifest
 [macrodef] creating macro  build-manifest
Trying to override old definition of task jarify
 [macrodef] creating macro  jarify
 [macrodef] creating macro  module-uptodate
 [macrodef] creating macro  compile-test-macro
Trying to override old definition of task test-macro
 [macrodef] creating macro  test-macro
[available] Unable to find file pom.xml to set property pom.xml.present
 [macrodef] creating macro  compile
 [macrodef] creating macro  invoke-javacc
Trying to override old definition of task invoke-javadoc
 [macrodef] creating macro  invoke-javadoc
 [macrodef] creating macro  contrib-crawl
 [macrodef] creating macro  svn-export-source
 [macrodef] creating macro  get-svn-info
 [macrodef] creating macro  make-checksums
 [macrodef] creating macro  sign-artifacts-macro
 [macrodef] creating macro  copy-to-stage-macro
      [ant] Entering /home/hossman/lucene/branch_lucene3930/lucene/tools/build.xml...
Build sequence for target(s) `compile-core' is [download-ivy, install-ivy, resolve, init, compile-core]
Complete build sequence is [download-ivy, install-ivy, resolve, init, compile-core, compile-tools, validate, common.init, common.download-java6-javadoc-packagelist, common.install-ivy, check-analyzers-common-uptodate, common.clover.check, common.jar-core, clover.setup, junit-sequential, clover.info, clover, common.compile-core, junit-mkdir, compile-test-framework, compile-test, check-lucene-core-uptodate, common.jflex-uptodate-check, jar-core, jar-src, javadocs, install-maven-tasks, common.dist-maven, common.junit-mkdir, common.clean, check-queries-uptodate, jar-queries, javacc-uptodate-check, common.generate-test-reports, rat-sources-typedef, rat-sources, jflex-uptodate-check, common.javadocs, jar, javacc-check, jflex-check, dist-maven, default, jflex-notice, common.jar-lucene-core, common.javacc-notice, common.compile-tools, common.jar-queries, common.jar-src, common.rat-sources, compile-lucene-core, download-java6-javadoc-packagelist, common.clover.setup, common.default, junit-parallel, test, common.clover, common.junit-sequential, jar-lucene-core, common.jflex-check, clover.check, common.check-queryparser-uptodate, common.compile-test, common.jar-analyzers-common, compile, clean, common.clover.info, common.validate, common.download-ivy, check-queryparser-uptodate, common.jar-queryparser, common.check-analyzers-common-uptodate, generate-test-reports, common.javacc-uptodate-check, common.resolve, common.javacc-check, common.install-maven-tasks, common.rat-sources-typedef, jar-queryparser, common.compile-test-framework, jar-analyzers-common, common.compile-lucene-core, common.junit-parallel, common.jar, common.jflex-notice, common.check-queries-uptodate, javacc-notice, common.compile, common.check-lucene-core-uptodate, common.test, ]

download-ivy:
Skipped because property 'ivy.available' set.

install-ivy:
parsing buildfile jar:file:/home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar!/org/apache/ivy/ant/antlib.xml with URI = jar:file:/home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar!/org/apache/ivy/ant/antlib.xml
Trying to override old definition of datatype antlib:org.apache.ivy.ant:settings
Trying to override old definition of task antlib:org.apache.ivy.ant:configure
Trying to override old definition of task antlib:org.apache.ivy.ant:resolve
Trying to override old definition of task antlib:org.apache.ivy.ant:retrieve
Trying to override old definition of task antlib:org.apache.ivy.ant:deliver
Trying to override old definition of task antlib:org.apache.ivy.ant:publish
Trying to override old definition of task antlib:org.apache.ivy.ant:extract
Trying to override old definition of task antlib:org.apache.ivy.ant:cachepath
Trying to override old definition of task antlib:org.apache.ivy.ant:cachefileset
Trying to override old definition of task antlib:org.apache.ivy.ant:report
Trying to override old definition of task antlib:org.apache.ivy.ant:repreport
Trying to override old definition of task antlib:org.apache.ivy.ant:var
Trying to override old definition of task antlib:org.apache.ivy.ant:check
Trying to override old definition of task antlib:org.apache.ivy.ant:artifactproperty
Trying to override old definition of task antlib:org.apache.ivy.ant:buildlist
Trying to override old definition of task antlib:org.apache.ivy.ant:install
Trying to override old definition of task antlib:org.apache.ivy.ant:convertpom
Trying to override old definition of task antlib:org.apache.ivy.ant:makepom
Trying to override old definition of task antlib:org.apache.ivy.ant:artifactreport
Trying to override old definition of task antlib:org.apache.ivy.ant:info
Trying to override old definition of task antlib:org.apache.ivy.ant:addpath
Trying to override old definition of task antlib:org.apache.ivy.ant:listmodules
Trying to override old definition of task antlib:org.apache.ivy.ant:findrevision
Trying to override old definition of task antlib:org.apache.ivy.ant:buildnumber
Trying to override old definition of task antlib:org.apache.ivy.ant:cleancache

resolve:
[ivy:retrieve] No ivy:settings found for the default reference 'ivy.instance'.  A default instance will be used
[ivy:retrieve] Loading jar:file:/home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar!/org/apache/ivy/core/settings/ivy.properties
[ivy:retrieve] searching settings file: trying /home/hossman/lucene/branch_lucene3930/lucene/tools/ivysettings.xml
[ivy:retrieve] searching settings file: trying /home/hossman/lucene/branch_lucene3930/lucene/tools/ivyconf.xml
[ivy:retrieve] searching settings file: trying ivysettings.xml
[ivy:retrieve] searching settings file: trying ivyconf.xml
[ivy:retrieve] no settings file found, using default...
[ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] jakarta commons httpclient not found: using jdk url handling
[ivy:retrieve] :: loading settings :: url = jar:file:/home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
[ivy:retrieve] no default ivy user dir defined: set to /home/hossman/.ivy2
[ivy:retrieve] including url: jar:file:/home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar!/org/apache/ivy/core/settings/ivysettings-public.xml
      [ant] Exiting /home/hossman/lucene/branch_lucene3930/lucene/tools/build.xml.
  [antcall] Exiting /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/build.xml.
   [subant] Exiting /home/hossman/lucene/branch_lucene3930/lucene/contrib/demo/build.xml.
   [subant] Exiting /home/hossman/lucene/branch_lucene3930/lucene/build.xml.
java.lang.OutOfMemoryError: PermGen space
    at org.apache.tools.ant.DefaultLogger.formatTime(DefaultLogger.java:323)
    at org.apache.tools.ant.DefaultLogger.buildFinished(DefaultLogger.java:170)
    at org.apache.tools.ant.Project.fireBuildFinished(Project.java:2037)
    at org.apache.tools.ant.Main.runBuild(Main.java:778)
    at org.apache.tools.ant.Main.startAnt(Main.java:217)
    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
    at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
PermGen space
asfimport commented 12 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

Attaching full stderr/stdout of a (differnet "ant -verbose clean test" run that ended with another PermGen OOM...

hossman@bester:\~/lucene/branch_lucene3930$ ant -verbose clean test 2>&1 > ant_-verbose_clean_test.out.txt
java.lang.OutOfMemoryError: PermGen space
    at java.lang.ClassLoader.findBootstrapClass(Native Method)
    at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:927)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:298)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at org.apache.tools.ant.DefaultLogger.formatTime(DefaultLogger.java:323)
    at org.apache.tools.ant.DefaultLogger.buildFinished(DefaultLogger.java:170)
    at org.apache.tools.ant.Project.fireBuildFinished(Project.java:2037)
    at org.apache.tools.ant.Main.runBuild(Main.java:778)
    at org.apache.tools.ant.Main.startAnt(Main.java:217)
    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
    at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
PermGen space
asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Now we have the problematic patched jars and the renamed unreleased jars.

For the unreleased jars, can we just include their source in our source? This would work for commons-csv and noggit which are the only two in solr/lib. They're both very small jars with only a handful of java files.

I'm running into bigger problems with the libs in the solr contrib langid. The langdect jar is from a revision I cannot find in its googlecode project (due to a switch from svn to git) and the jsonic jar it depends on isn't available in ivy accessible repos, only version 1.2.8 is.

asfimport commented 12 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

as far as the PermGen OOM goes, the -verbose logs show repeated instances of "Trying to override old definition of task antlib:org.apache.ivy.ant:..." suggesting that rmuir's unless="ivy.uptodate" isn't working the way we think it should (possibly because of the way the various build files are included in one another?)

If we can't keep the <taskdef/ from running multiple times, we might be able to force it to use the same class loader each time, per the ant docs... https://ant.apache.org/manual/Tasks/typedef.html

If you are defining tasks or types that share the same classpath with multiple taskdef or typedef tasks, the corresponding classes will be loaded by different Java ClassLoaders. Two classes with the same name loaded via different ClassLoaders are not the same class from the point of view of the Java VM, they don't share static variables and instances of these classes can't access private methods or attributes of instances defined by "the other class" of the same name. They don't even belong to the same Java package and can't access package private code, either.

The best way to load several tasks/types that are supposed to cooperate with each other via shared Java code is to use the resource attribute and an antlib descriptor. If this is not possible, the second best option is to use the loaderref attribute and specify the same name for each and every typedef/taskdef - this way the classes will share the same ClassLoader. Note that the typedef/taskdef tasks must use identical classpath defintions (this includes the order of path components) for the loaderref attribute to work.

...it appears it's just some unique string key to name the classloader? (no idea if it whatever is causing hte current problem will still plague by creating multiple loaders with the same name)... https://svn.apache.org/viewvc/ant/core/tags/ANT_171/src/etc/testcases/core/loaderref/

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Proof of concept patch to show how we can download jetty, patch it, build it and copy the jars of interest into lib.

I've made this against the 3930 branch but its targeting 3x of course.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

as far as the PermGen OOM goes, the -verbose logs show repeated instances of "Trying to override old definition of task antlib:org.apache.ivy.ant:..." suggesting that rmuir's unless="ivy.uptodate" isn't working the way we think it should (possibly because of the way the various build files are included in one another?)

Thats right... I don't have a good solution here yet.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I read up enough on that issue to know I don't want to be screwing around with it.

So you have to install ivy in your \~/.ant/lib: I removed any automagical shit, and this is now solved.

asfimport commented 12 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

So you have to install ivy in your \~/.ant/lib

I personally don't like it when I need to install ant dependencies in a global scope – this may not be a problem from one project's perspective but if you're working on multiple projects then this can result in global dependencies shadowing project's local definitions and debugging this is a pain.

Not to mention it's another requirement after checkout. I won't be able to look into this now, just expressing my opinion.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Proof of concept patch to show how we can download jetty, patch it, build it and copy the jars of interest into lib.

Yeah, thats how we should do it. Long term, for developers (like committers) that do this every day, we can add a property you can define in your \~/build.properties declaring where a patched jar already exists so it doesn't slow us down: but thats an optimization for guys like us.

I'm running into bigger problems with the libs in the solr contrib langid. The langdect jar is from a revision I cannot find in its googlecode project (due to a switch from svn to git) and the jsonic jar it depends on isn't available in ivy accessible repos, only version 1.2.8 is.

Ouch, i didn't notice they changed over to git... The git repository they have just doesn't correspond to any meaningful rev numbers (die git die), but still has the history from svn. We are going to have to do a git checkout i think?

asfimport commented 12 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

\~/build.properties

Again – can we make it project-wide (${project.home}/build.properties) or at least have a unique name (\~/lucene-solr.properties)?

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I personally don't like it when I need to install ant dependencies in a global scope – this may not be a problem from one project's perspective but if you're working on multiple projects then this can result in global dependencies shadowing project's local definitions and debugging this is a pain.

Not to mention it's another requirement after checkout. I won't be able to look into this now, just expressing my opinion.

There's nothing I can do about this. I spent all day trying to look for a better solution. You can use ant -lib too if you want, nothing stops you from doing that.

But we have to remove all these jars, and for the build to still work. I'm sure ill piss a lot of people off.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Again – can we make it project-wide (${project.home}/build.properties) or at least have a unique name (\~/lucene-solr.properties)?

its always been:

  <!-- Give user a chance to override without editing this file
      (and without typing -D each time it compiles it -->
  <property file="${user.home}/lucene.build.properties"/>
  <property file="${user.home}/build.properties"/>
  <property file="${basedir}/build.properties"/>
  <property file="${common.dir}/build.properties"/>

So just use lucene.build.properties if you want that.

asfimport commented 12 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

You can use ant -lib too if you want, nothing stops you from doing that.

Sure, but this introduces additional complexity in setting up shell aliases or simply remembering about it. This just feels wrong somehow (making life harder instead of simpler). I may take a look at it at some point but I also remember having difficulties with multiple antlib definitions in our stuff and I believe you it's a pain.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Sure, but this introduces additional complexity in setting up shell aliases or simply remembering about it. This just feels wrong somehow (making life harder instead of simpler). I may take a look at it at some point but I also remember having difficulties with multiple antlib definitions in our stuff and I believe you it's a pain.

Right but honestly, while it might not be ideal, I don't think its a blocker? Having jars in our source release is. Having the build OOM is. So I made a tradeoff.

If you (or any one else) has the time to look at stuff like this, it would be really appreciated: thats why i created a branch:

https://svn.apache.org/repos/asf/lucene/dev/branches/lucene3930

We likely also won't have any of the IDE configurations working unless someone has the time to jump into that, and I have no idea if the maven build will break by us not having jars in the source tree. These are all things I will ignore without hesitation because they don't need to block a release. If anyone wants to fix this stuff, just start committing to the branch, i need your help... its just that simple :)

asfimport commented 12 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Having jars in our source release is.

Is a requirement that the source release must "build out of the box"? Or is it about code publication only? I don't know, I'm asking. This seems like a revolutionary build change before the release :)

Having the build OOM is. So I made a tradeoff.

I understand this, but my gut feeling still says that if you need to install ivy in an ant global space you might as well set ANT_OPTS to increase permgen... The tradeoff made is one of many.

i need your help... its just that simple

Can't jump into it right now, sorry. I'll take a look when I get a spare cycle though. I'm not sure it can be fixed but I'll take a look.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

Is a requirement that the source release must "build out of the box"? Or is it about code publication only? I don't know, I'm asking. This seems like a revolutionary build change before the release

Yeah you are telling me. trust me i'm not happy. see my note to the dev-list:

So i didnt just rush into this shit and make it blocker as some knee-jerk reaction. I don't feel i have any other choice.

As i said on my email, given the discussion about this on the other thread, i'd rather fix shit than argue on mailing lists about whether our release is legal or not. If anyone else has a different opinion and is willing to sacrifice days of their life arguing with apache board members instead, then stand up and we'll go with whats in branch_3x now.

But given the discussion, personally I'm not going to be involved in (and will vote against) any release that has jar files in the source.tgz.

asfimport commented 12 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Clear. It's a pity we have to deal with it right before the release but I understand (or rather accept) the rationale.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

FWIW.... i did a completley clean checkout of the lucene3930 (r1306662) and got the following build failure trying to run "ant clean test" from the top level.

Hoss, thank you for checking out the branch and doing tests! this is really useful, i know there are more nasties hanging out there...

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

For the unreleased jars, can we just include their source in our source? This would work for commons-csv and noggit which are the only two in solr/lib. They're both very small jars with only a handful of java files.

Any opinions on this? Do I need to change the packages for anything that we put into our source?

Ouch, i didn't notice they changed over to git... The git repository they have just doesn't correspond to any meaningful rev numbers (die git die), but still has the history from svn. We are going to have to do a git checkout i think?

A checkout of head and upgrade to use that? or do a checkout of an older revision? We're going to have to checkout the jsonic source as well.

We likely also won't have any of the IDE configurations working unless someone has the time to jump into that, and I have no idea if the maven build will break by us not having jars in the source tree. These are all things I will ignore without hesitation because they don't need to block a release. If anyone wants to fix this stuff, just start committing to the branch, i need your help... its just that simple

I think both IDE support and Maven will be fine. The idea and eclipse targets will need to invoke resolve to ensure the libs are downloaded. Once they're in the same place as they were before, the IDE configurations should march along fine. Any libs we've changed the name of or dropped, we can just update the configs for. For Maven all this work is actually going to simplify the poms. We just need to bring the dependencies inline with the libs we have. Both wont be big deals.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

A checkout of head and upgrade to use that? or do a checkout of an older revision? We're going to have to checkout the jsonic source as well.

I don't care as long as it works, and as long as its not 'head' but a specific revision/hashcode/whatever so its not changing...

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

And for reference, the reason we used the revision we had for that, is so that we would load the language profiles from the classpath and not the filesystem.

In other words, we need this change: http://code.google.com/p/language-detection/issues/detail?id=24

asfimport commented 12 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

For git the revision md5 is unique and you can always do a checkout of a particular revision (typically using so-called detached head). This just moves you to a particular version in the revision tree.

asfimport commented 12 years ago

Chris Male (migrated from JIRA)

Patch which places noggit and commons csv under org.apache.solr.internal.* and updates everything.

Will definitely need to include a package.html for the internal package detailing its role.

asfimport commented 12 years ago

Robert Muir (@rmuir) (migrated from JIRA)

I think its not ideal, but it nukes the jars, so lets do it. I can't think of a solution that really is ideal there...

asfimport commented 12 years ago

Yonik Seeley (@yonik) (migrated from JIRA)

Ugh... We copied the source for commons-csv and noggit? Feels like the cure is worse than the disease here...