search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.65k stars 1.03k forks source link

Make the Lucene jar an OSGi bundle [LUCENE-1344] #2421

Closed asfimport closed 8 years ago

asfimport commented 16 years ago

In order to use Lucene in an OSGi environment, some additional headers are needed in the manifest of the jar. As Lucene has no dependency, it is pretty straight forward and it ill be easy to maintain I think.

Migrated from LUCENE-1344 by Nicolas Lalevée, 12 votes, resolved Oct 25 2016 Attachments: lucene_trunk.patch, LUCENE-1344.patch (versions: 6), LUCENE-1344-3.0-branch.patch, LUCENE-1344-maven.patch, LUCENE-1344-r679133.patch, LUCENE-1344-r690675.patch, LUCENE-1344-r690691.patch, LUCENE-1344-r696747.patch, MANIFEST.MF.diff Linked issues:

asfimport commented 16 years ago

Nicolas Lalevée (migrated from JIRA)

Here is a patch against trunk.

The patch on common-build.xml allows the core or every contrib to package the jar as an OSGi bundle. When building, it just need a property "bundle.manifest.file" which is pointing to the template MANIFEST.MF to use. The patch on build.xml with the MANIFEST.MF makes lucene core jar an OSGi bundle.

Also in order to not have to maintain a third version scheme, I have added a /release target to compute the versions. So this doc should be updated : http://wiki.apache.org/lucene-java/ReleaseTodo

ant -Dversion=2.3.0-rc1 -Dspec.version=2.3.0 clean dist dist-src generate-maven-artifacts

should be replaced by:

ant /release clean dist dist-src generate-maven-artifacts

Then about maintenance, the version in the MANIFEST.MF file is just usefull for people having the Lucene source in Eclipse and usin it as an OSGi bundle. The version is actually overridden while building the jar. And every new java package that is part of the Lucene API have to be added to the Export-Package header.

asfimport commented 16 years ago

Nicolas Lalevée (migrated from JIRA)

Updated patch, which fix the Bundle-RequiredExecutionEnvironment, and now the packages are exported with their version

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Should we do this for 2.4? Nicolas, there are quite a few changes to common-build.xml – do you see any risks/downsides to these changes?

asfimport commented 16 years ago

Nicolas Lalevée (migrated from JIRA)

You are right to ask Michael, one part of the patch was risky, the part that was computing the version dynamically. I changed the way the properties were loaded and it made some mess with building the contrib during the release process. So I removed it. So then instead of 

ant -Dversion=2.3.0-rc1 -Dspec.version=2.3.0 clean dist dist-src generate-maven-artifacts

there should be

ant -Dversion=2.3.0-rc1 -Dspec.version=2.3.0 -Dbundle.version=2.3.0.cr1 clean dist dist-src generate-maven-artifacts

Note that there is no spell mistake here. The OSGi specification expects that the qualifier part of the version (the last one) is incremented based on the lexical order. So 2.3.0 < 2.3.0.rc1. But 2.3.0.alpha1 < 2.3.0.beta1 < 2.3.0.beta2 < 2.3.0.cr1 < 2.3.0.final

About the remaining patch, it is really just about adding additional fields into the manifest.mf, addition which is triggered by the existence of the bundle.manifest.file property. I attached the diff of the manifest of the core of a 2.4.0 release, between the trunk version and the patched one. And If I remove the property bundle.manifest.file from the build.xml, there is no diff at all (apart from the Implementation-Version because of the timestamp).

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Nicolas, does this mean we need to maintain the new META-INF/MANIFEST.MF, manually? Ie on each release go edit it & bump up the versions in there? (We need to update the wiki to this effect too)

There are quite a few "copies" of version in there, and for example we have Bundle-Version: 2.4.0.dev and then Specification-Version: 2.4.0-dev – is that right? (Ie, one uses "." and the other uses "-").

I was able to generate a release candidate using your command above. What command would be used to generate the actual release? (Ie, what to specify for version, spec.version and bundle.version?).

So 2.3.0 < 2.3.0.rc1. But 2.3.0.alpha1 < 2.3.0.beta1 < 2.3.0.beta2 < 2.3.0.cr1 < 2.3.0.final

OK I'm a little confused by this. What does "cr1" mean? And, while I can see the lexicographic sort order, your first case (2.3.0 < 2.3.0rc1) seems backwards because 2.3.0rc1 ("release candidate 1") arrives before 2.3.0 (this is the released 2.3.0 right?) in time, whereas in the 2nd case those releases are in time order. I'm confused.

Sorry for the silly questions – I know very little about OSGi bundles!

asfimport commented 16 years ago

Nicolas Lalevée (migrated from JIRA)

Nicolas, does this mean we need to maintain the new META-INF/MANIFEST.MF, manually? Ie on each release go edit it & bump up the versions in there? (We need to update the wiki to this effect too)

in the META-INF/MANIFEST.MF, only the Export-Package header is important to maintain. The other headers will never change or will be overrided by the build system. The other headers are also there for the users (aka me) that have imported the Lucene source in Eclipse and are willing to use it as an OSGi bundle.

There are quite a few "copies" of version in there, and for example we have Bundle-Version: 2.4.0.dev and then Specification-Version: 2.4.0-dev - is that right? (Ie, one uses "." and the other uses "-").

About the different version schemes, yep, this is yet another one to maintain. The version number taken into account in a OSGI environment is "Bundle-Version", I don't know what the header "Specification-Version" is used for. I tried to refactor a little bit in the build system to generate the version numbers, but I failed, a more bigger patch would be needed (I am willing to do some if needed).

I was able to generate a release candidate using your command above. What command would be used to generate the actual release? (Ie, what to specify for version, spec.version and bundle.version?).

It will be just about adding the OSGi version to the current used command line:

ant -Dversion=2.3.0 -Dbundle.version=2.3.0.final clean dist dist-src generate-maven-artifacts

OK I'm a little confused by this. What does "cr1" mean?

it would mean "candidate release 1". Not very english but it is the best we found when discussing about it for Ivy.

And, while I can see the lexicographic sort order, your first case (2.3.0 < 2.3.0rc1) seems backwards because 2.3.0rc1 ("release candidate 1") arrives before 2.3.0 (this is the released 2.3.0 right?) in time, whereas in the 2nd case those releases are in time order. I'm confused.

That was exactly my point. Due to the way OSGi orders the versions, if the release candicate version number is 2.3.0.rc1, and the final version is 2.3.0.final, then OSGi will understand that 2.3.0.final is previous to 2.3.0.rc1, which is wrong. In the second case, if the release candidate number is 2.3.0.cr1, then OSGI will be right and will order the different Lucene version as expected. You could also choose 2.3.0.rc1 and 2.3.0.zfinal but then it is a question of taste :)

Sorry for the silly questions - I know very little about OSGi bundles!

your questions are welcomed Michael ;)

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Thanks Nicolas. I understand a bit more now :)

One problem: even though I was able to successfully run the above command, the resulting MANIFEST.MF in the Lucene core JAR (dist/maven/org/apache/lucene/lucene-core/2.3.0/lucene-core-2.3.0.jar) does not have any of your added lines (eg Export-Package) – do you see this too?

About the different version schemes, yep, this is yet another one to maintain. The version number taken into account in a OSGI environment is "Bundle-Version", I don't know what the header "Specification-Version" is used for. I tried to refactor a little bit in the build system to generate the version numbers, but I failed, a more bigger patch would be needed (I am willing to do some if needed).

I think it's OK for now if we have to update the versions in META-INF/MANIFEST.MF manually as part of the release process? (It sounds hard to get the build to autogen the versions).

asfimport commented 16 years ago

Nicolas Lalevée (migrated from JIRA)

About the missing header in the maven jar, this is weird because they exist in every other jar in the distrib but in the maven one. And this is a lot more strange to see that the manifest of the lucene core jar is in fact the manifest of the demo one... And I retested without the patch, everything works correctly. I don't see yet how it can happen.

And the META-INF/MANIFEST.MF file doesn't have to be updated when releasing. The build process is overriding the header entries. The file is mainly a template. See the build-bundle-manifest macro in the patch. But it will have to be updated after the release, just like the common-build.xml, to update the version number.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

About the missing header in the maven jar, this is weird because they exist in every other jar in the distrib but in the maven one. And this is a lot more strange to see that the manifest of the lucene core jar is in fact the manifest of the demo one... And I retested without the patch, everything works correctly. I don't see yet how it can happen.

Actually I can't find "Export-Package" header in any of the jars under dist/maven/* (at least, contrib/wordnet, contrib/highlighter and contrib/benchmark).

I'll hold off committing until you figure this out :)

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

I would love to see bundle headers in the manifests.

RE: 2.3.0 < 2.3.0.rc1} One way to handle this problem is byI using a timestamp as qualifier when generating the version number. This ensures that each new build is of a higher version. At Eclipse we use a ".qualifier" suffix which will be replaced with the actual CVS/SVN tag at build time. The CVS/SVN tag is actually a timestamp "v20080916-1020" or just an increasing number (eg., "v1, v2, v3..."). It depends on what a project favors. I could also imaging that this would be the SVN revision at build time.

asfimport commented 16 years ago

Nicolas Lalevée (migrated from JIRA)

This is a good idea as soon as you can be sure that you are building the different versions in the correct order. I think that in most use case, it will work. But it seems wrong to bind the versions order to the builds order.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

But it seems wrong to bind the versions order to the builds order.

Not all the versions ... just those which appear to be of the same version (i.e., same tag or branch). If it's a different version, you would change the numbers before the qualifier. The qualifier really is just a build identifier. Giving it a meaning full name like .rc1 is a nice thing to have. But in reality this is hard to manage with automatic provisioning/update systems. They usually handle automatic updates to a higher build of the same version or a different build of a higher version fine.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Given the healthy discussion here, plus the fact that the current patch doesn't actually add Export-Package to the manifest, and this is the last bug holding up 2.4, I think we should take this off the plate for 2.4.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

Michael, when would be the last chance to submit a patch for inclusion into 2.4? I really, really need this because I'm about to embed Lucene into an OSGi based solution and this would really help.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

The current plan is to branch tomorrow, freeze the branch for \~10 days, and then release 2.4. If we can get a patch that works here say in the next few days we might be able to get it into 2.4?

If/when we do 2.4.1 we could probably backport this into that as well (since it's a small change).

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

Michael, attached is a patch which is less intrusive and mimics what Lucene does for Maven. It adds a new target to the build.xml called generate-osgi-bundles. This generated the OSGi bundles into dist/osgi (similar to maven). It uses Peter Kriens Bnd Tool to generate the bundle manifest. Thus, no additional work is involved for you guys.

Please review and comment. Feel free to commit meanwhile. I'll continue to work on OSGi-ifing the other JARs as well. However, this is a bit more tricky because it requires a correct classpath setup and I haven't fully got into how you build contrib jars.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

This patch replaces the previous one and implements building OSGi JARS for all contribution. It follows the Maven build, i.e. an OSGi jar will only be created if an osgi.bnd.template file exists. I included one for the analyzers. Others can be contributed individually.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Thanks Gunnar; I'll try this today. What is the suggested ant command-line for building the dist now?

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

{{{ ant -Dversion=2.4.0 clean dist dist-src generate-maven-artifacts generate-osgi-artifacts }}}

It's important to note, that "spec.version" must always be major.minor.service (digits only). The qualifier is automatically computed based on the time stamp.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Gunnar, when I apply the patch and run a toplevel "ant clean" followed by "ant test", I hit an error when trying to compile/test contrib/highlighter. Are you seeing this too?

compile-highlighter:
     [echo] Building highlighter...

build-memory:
     [echo] Highlighter building dependency /lucene/lucene.osgi/build/contrib/memory/lucene-memory-2.4-dev.jar
     [echo] Building memory...

javacc-uptodate-check:

javacc-notice:

jflex-uptodate-check:

jflex-notice:

common.init:

build-lucene:

init:

clover.setup:

clover.info:
     [echo] 
     [echo]       Clover not found. Code coverage reports disabled.
     [echo]     

clover:

compile-core:
    [mkdir] Created dir: /lucene/lucene.osgi/build/contrib/highlighter/classes/java
    [javac] Compiling 22 source files to /lucene/lucene.osgi/build/contrib/highlighter/classes/java
    [javac] /lucene/lucene.osgi/contrib/highlighter/src/java/org/apache/lucene/search/highlight/WeightedSpanTermExtractor.java˘:35: package org.apache.lucene.index.memory does not exist
    [javac] import org.apache.lucene.index.memory.MemoryIndex;
    [javac]                                      ^
    [javac] /lucene/lucene.osgi/contrib/highlighter/src/java/org/apache/lucene/search/highlight/WeightedSpanTermExtractor.java:312: cannot find symbol
    [javac] symbol  : class MemoryIndex
    [javac] location: class org.apache.lucene.search.highlight.WeightedSpanTermExtractor
    [javac]       MemoryIndex indexer = new MemoryIndex();
    [javac]       ^
    [javac] /lucene/lucene.osgi/contrib/highlighter/src/java/org/apache/lucene/search/highlight/WeightedSpanTermExtractor.java:312: cannot find symbol
    [javac] symbol  : class MemoryIndex
    [javac] location: class org.apache.lucene.search.highlight.WeightedSpanTermExtractor
    [javac]       MemoryIndex indexer = new MemoryIndex();
    [javac]                                 ^
    [javac] 3 errors

BUILD FAILED
/lucene/lucene.osgi/build.xml:562: The following error occurred while executing this line:
/lucene/lucene.osgi/build.xml:552: The following error occurred while executing this line:
/lucene/lucene.osgi/contrib/benchmark/build.xml:173: The following error occurred while executing this line:
/lucene/lucene.osgi/contrib/highlighter/build.xml:41: The following error occurred while executing this line:
/lucene/lucene.osgi/common-build.xml:229: The following error occurred while executing this line:
/lucene/lucene.osgi/common-build.xml:505: Compile failed; see the compiler error output for details.
asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

I haven't tried this sequence but it looks like a general compile error.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

I haven't tried this sequence but it looks like a general compile error.

Right, but a clean checkout runs fine, so it's something in the patch that somehow caused highlighter to no longer see the lucene-memory-2.4-dev.jar I think.

asfimport commented 16 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

I think there is an unresolved issue with this anyway...I can't remember exactly, but I think one target that should work does not because it doesn't cooperate with the memory stuff (even though the overall build target does work). This may be another manifestation of that issue.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Hmm – if I revert this part of the patch:

`@@` -218,7 +225,7 `@@`
   <target name="compile-core" depends="init, clover"
           description="Compiles core classes">
     <compile
-      srcdir="src/java"
+      srcdir="${src.dir}"
       destdir="${build.dir}/classes/java">
       <classpath refid="classpath"/>
     </compile>

Then I'm able to "ant clean" and then "ant test" just fine. I don't yet understand why that fixes it, though.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

Here is an updated patch which adds OSGi templates for highlighter and memory as well.

I did a test using ant -Dversion=2.4.0 clean test dist dist-src generate-osgi-bundles. However, it fails because of some failing tests. I also noticed some OutOfMemoryErrors.

If I just run ant -Dversion=2.4.0 clean generate-osgi-bundles it doesn't compile and quits with the error you mentioned. However, that's in the build file area which I didn't touch.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

Ahh .. that makes sense. So it's caused by the patch. I changed it for consistency reasons but I guess it's better to revert that. It's related to how Ant resolves paths and overwrites/inherits properties.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

Here is the latest patch which reverts that. Now it compiles fine after a clean.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

Michael, I solved the issue with ${src.dir}.

The property is initially set the following way: {{{ <property name="src.dir" location="src/java"/> }}}

This resolves the path and sets the property to the actually path. It is then inherited to all sub ant calls. Thus, whenever you refer to ${src.dir} it actually was already set to the outer most. Thus, when compile-core is invoked it actually compiles highlight instead of memory.

If the property is defined the following way, it will resolve at reference time and not at definition time. {{{ <property name="src.dir" value="src/java"/> }}}

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Super, I'll make that change locally and test.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

OK I was able to generate the OSGI bundles. But, a few questions.

Under dist/osgi I see these files:

-rw-rw-rw-  1 mike  staff    5408 Sep 18 12:08 org.apache.lucene.core_2.4.0.200809181201.jar
-rw-rw-rw-  1 mike  staff  770986 Sep 18 12:08 org.apache.lucene.core.source_2.4.0.200809181201.jar
-rw-rw-rw-  1 mike  staff    5412 Sep 18 12:08 org.apache.lucene.analyzers_2.4.0.200809181208.jar
-rw-rw-rw-  1 mike  staff  431926 Sep 18 12:08 org.apache.lucene.analyzers.source_2.4.0.200809181208.jar
-rw-rw-rw-  1 mike  staff    5410 Sep 18 12:08 org.apache.lucene.memory_2.4.0.200809181208.jar
-rw-rw-rw-  1 mike  staff   37197 Sep 18 12:08 org.apache.lucene.memory.source_2.4.0.200809181208.jar
-rw-rw-rw-  1 mike  staff    5414 Sep 18 12:08 org.apache.lucene.highlighter_2.4.0.200809181208.jar
-rw-rw-rw-  1 mike  staff   45676 Sep 18 12:08 org.apache.lucene.highlighter.source_2.4.0.200809181208.jar

Shouldn't we make OSGI bundles for all contrib packages, and core & demo?

The source jar actually has .java sources, but the binary jars only have META-INF subdir, eg org.apache.lucene.core_2.4.0.200809181201.jar unzips to:

Archive:  dist/osgi/org.apache.lucene.core_2.4.0.200809181201.jar
  inflating: META-INF/MANIFEST.MF    
  inflating: META-INF/LICENSE.txt    
  inflating: META-INF/NOTICE.txt

Is that right? Shouldn't this jar have the .class files?

At a higher level, can't we have the normal JARs produced by the build simply include the necessary OSGI headers in the manifest? (Why make new JARs?).

Finally, during the release process, what should we do with the OSGI bundles? EG the Maven artifacts are copied to a distribution directory so that they are pushed to the central Maven repositories.

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

One more confusion on my part. The manifest for the core source jar (org.apache.lucene.core.source_2.4.0.200809181201.jar) is this:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.0
Created-By: 1.6.0_05-b13-52 (Apple Inc.)
Bundle-SymbolicName: org.apache.lucene.core.source
Bundle-ManifestVersion: 2
Bundle-Name: Lucene Search Engine: core source
Bundle-Version: 2.4.0.200809181201
Eclipse-SourceBundle: org.apache.lucene.core;version="2.4.0.2008091812
 01"
Bundle-Vendor: The Apache Software Foundation
Specification-Title: Lucene Search Engine: core
Specification-Version: 2.4.0
Specification-Vendor: The Apache Software Foundation
Implementation-Title: org.apache.lucene
Implementation-Version: 2.4.0 696661M - 2008-09-18 12:01:47
Implementation-Vendor: The Apache Software Foundation
X-Compile-Source-JDK: 1.4
X-Compile-Target-JDK: 1.4

(Note the newline inside the timestamp on the Eclipse-SourceBundle – is that OK?).

Whereas the manifest for the binary jar (org.apache.lucene.core_2.4.0.200809181201.jar) is much smaller:

Manifest-Version: 1.0
Bundle-Vendor: The Apache Software Foundation
Bundle-Version: 2.4.0.200809181201
Bundle-Name: Lucene Search Engine: core
Bundle-ManifestVersion: 2
Bundle-License: http://www.apache.org/licenses/LICENSE-2.0
Bundle-SymbolicName: org.apache.lucene.core
Bundle-RequiredExecutionEnvironment: J2SE-1.5,J2SE-1.4

Is that expected? And the manifest for the source jar is missing at least Bundle-License?

asfimport commented 16 years ago

Nicolas Lalevée (migrated from JIRA)

I have fixed my patch so the maven jar does contain the OSGi manifest, even if I don't understand really why the jar are generated twice. This new patch contains improvements.

So the command to build a rc version will be:

ant -Dspec.version=2.4.0 -Dqualifier.version=rc1 clean dist dist-src generate-maven-artifacts

will generate:

For a final version:

ant -Dspec.version=2.4.0 -Dqualifier.version=final clean dist dist-src generate-maven-artifacts

will generate:

I know that Lucene distrib used to not add qualifier to the final release, it can still be done with:

ant -Dspec.version=2.4.0 -Dqualifier.version=final -Dversion=2.4.0 clean dist dist-src generate-maven-artifacts

will generate:

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

Michael, the attached patch adds the bundle license header for the source bundles. I also started to add descriptors for two more contrib jars (queries and xml parser). It gets a bit messy now because of split-packages, i.e. the queries jar exports the same package (org.apache.lucene.search) as Lucene core. This requires to add split packages headers to the manifests. Additional, the "-" (dash) is invalid so I had to override the default bundle symbolic name for the xml-query-parser jar. I use the package name.

The OSGi JARs don't need to be uploaded somewhere. It's enough to just put them on the FTP mirrors next to the regular distribution.

Nicholas, we generate different JARs for OSGi because I don't like to mix and match things. OSGi should be separated from Maven. They are different. For example, in OSGi it's a good practice to name the JAR like the bundle symbolic name and to include the version. Otherwise you'll run into problems because bundle repositories are usually not hierarchically organized like Maven repos.

Michael, do we need a "-final" or "-dev" indicator on the JAR names and bundle versions?

asfimport commented 16 years ago

Nicolas Lalevée (migrated from JIRA)

Nicholas, we generate different JARs for OSGi because I don't like to mix and match things. OSGi should be separated from Maven.

Maven is about building and resolving dependencies, OSGi is about a running environement. And they can be mixed with no worry, it even exists a maven plugin to handle OSGi headers, and a maven repository full of OSGi bundles.

They are different. For example, in OSGi it's a good practice to name the JAR like the bundle symbolic name and to include the version. Otherwise you'll run into problems because bundle repositories are usually not hierarchically organized like Maven repos.

I don't think the purpose here is to make Lucene have its own OSGi repository. And nothing here prevents you to build a pure OSGi repository with those jars with the name convention you like.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

I've updated the patch to include more descriptors for other contrib jars. Should be complete now (except for the ant libs, benchmarks and bdb which can be added later if necessary).

It's also possible to set the OSGi qualifier using property osgi.version.qualifier. The default is .yyyyMMddHHmm

asfimport commented 16 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Thanks Gunnar.

Can you and Nicolas work together to reconcile your two different patches here? I really know very little about the specifics involved here and am involved more or less as a "catalyst" to try to move things along... thanks.

asfimport commented 16 years ago

Gunnar Wagenknecht (migrated from JIRA)

I'll collect feedback on the developers list on what way they prefer.

asfimport commented 14 years ago

Enrico Schnepel (migrated from JIRA)

Is there any progress on this issue?

asfimport commented 13 years ago

Kirby Bohling (migrated from JIRA)

Attaching a patch that generates a sibling jar file (hopefully it will replace the original Jar files before it is actually committed) that includes all of the OSGi Manifest requirements.

All the regular unit tests appear to pass if I use the OSGi bundles instead of the standard jar files (by removing the <jar> task in the <jarify> macro defined in commons-build.xml, and modifying the <bnd> task to use output="@{destfile}". The backward compat tests appear to fail, but I haven't gotten to the bottom of that just yet.

It requires that the biz.aQute.bnd.jar be in the lib/ directory.

From this site: http://www.aqute.biz/Bnd/Download

I grabbed it off of this link: http://dl.dropbox.com/u/2590603/bnd/biz.aQute.bnd.jar

The bnd jar is a standard tool for generating valid and up-to-date OSGi headers with minimal problems.

I downloaded the commons-math jar file to see the set of headers the commons projects added to their bundles.

I have not tested any of the contrib bundles in an OSGi environment, but I am using the Lucene Core bundle generated by this. It is also likely that the ${Name} value should be set in all of the sub-directories to generate better human consumable names for the bundles, currently they are all set to "Lucene". Potentially a better description would be useful also. I believe a couple will have problems (lucli being the most obvious because it isn't in the org.apache.lucene package structure). I can do more testing and ensure as much as I can works if this patch would be applied.

I can port and update this to the 4.0 branch if desirable, and it should be simple to modify to just generate the final Jar files.

asfimport commented 13 years ago

Luca Stancapiano (migrated from JIRA)

I'm interested to this feauture too. Is it committed?

asfimport commented 13 years ago

Ryan McKinley (@ryantxu) (migrated from JIRA)

I have not followed this patch and don't know how it works... but this may be pretty easy to add to the (officially unsupported) maven build:

          <plugin>
              <groupId>org.apache.felix</groupId>
              <artifactId>maven-bundle-plugin</artifactId>
              <version>1.4.0</version>
              <extensions>true</extensions>
              <configuration>
                  <instructions>
                      <Bundle-SymbolicName>${pom.groupId}.${pom.artifactId}</Bundle-SymbolicName>
                      <Bundle-Name>${pom.name}</Bundle-Name>
                      <Bundle-Version>${pom.version}</Bundle-Version>
                      <Bundle-Activator>org.wso2.mbp.sample01.Activator</Bundle-Activator>
                      <Private-Package>org.wso2.mbp.sample01</Private-Package>
                  </instructions>
              </configuration>
          </plugin>

http://wso2.org/library/tutorials/develop-osgi-bundles-using-maven-bundle-plugin

asfimport commented 13 years ago

Daniele (migrated from JIRA)

The solution is almost that one, just a few corrections:

<plugin> <groupId>org.apache.felix</groupId> <artifactId>maven-bundle-plugin</artifactId> <version>2.3.4</version> <extensions>true</extensions> <configuration> <instructions> <Bundle-SymbolicName>${project.groupId}.${project.artifactId}</Bundle-SymbolicName> <Bundle-Name>${project.name}</Bundle-Name> <Bundle-Version>${project.version}</Bundle-Version> </instructions> </configuration> </plugin>

I'd also add a executions block (direct child of the plugin block)

      &lt;executions&gt;
        &lt;execution&gt;
          &lt;id&gt;bundle-manifest&lt;/id&gt;
          &lt;phase&gt;process-classes&lt;/phase&gt;
          &lt;goals&gt;
            &lt;goal&gt;manifest&lt;/goal&gt;
          &lt;/goals&gt;
        &lt;/execution&gt;
      &lt;/executions&gt;

so that is not needed to change the packaging type to bundle to make the MANIFEST information to be attached.

asfimport commented 13 years ago

Ryan McKinley (@ryantxu) (migrated from JIRA)

Thanks Daniele – can you make a patch for this?

I don't use OSGi so I have no idea if the results are valid or not, but would happily commit changes to the maven poms if someone verifies the changes work.

asfimport commented 13 years ago

Luca Stancapiano (migrated from JIRA)

I don't see maven in the lucene sources. It is built with ant. The first solution seems ok....why it was not committed?

asfimport commented 13 years ago

Ryan McKinley (@ryantxu) (migrated from JIRA)

Hi Luca – maven support is in the dev-tools folder: https://svn.apache.org/repos/asf/lucene/dev/trunk/dev-tools/maven

My understanding is that OSGi needs to know about the dependencies – i don't know how (or if) our ant build can support that.

asfimport commented 13 years ago

Daniele (migrated from JIRA)

oh, well, the maven plugin just automatically generates the Manifest.mf information bases on the actual code base. It works nice but is not the only solution. You can run http://bndtools.org/ against the jar to make a new jar with correct Manifest OSGi instructions, that's what I did to make the 3.1.0 release work on an OSGi environment. I think you can also integrate bnd with an ant script. Or you can just keep the source Manifest file up to date, but you have to keep exported packages (and version) aligned manually.

In the end is just a matter of releasing a jar with a few more meta information in the standard Manifest.MF file, nothing more.

asfimport commented 13 years ago

Ryan Hill (migrated from JIRA)

The bndtools approach works for me – I believe that is that the Jackson (JSON) project does as well.

asfimport commented 13 years ago

Luca Stancapiano (migrated from JIRA)

Ok ....I created a patch according the pom configuration seen before. It creates osgi bundles for lucene-core and the other modules. I tested them inside felix and they are installed correctly. There are not mainteinance problems because the packages are generated dinamically by the maven-bundle-plugin configured in this patch

asfimport commented 13 years ago

Ryan McKinley (@ryantxu) (migrated from JIRA)

Thanks Luca!

When you upload a patch, can you click the "Grant license to ASF for inclusion in ASF works" button? It seems a little silly for this small patch, but that is ASF policy.

thanks

asfimport commented 13 years ago

Ryan McKinley (@ryantxu) (migrated from JIRA)

This is the same patch, but moved to the lucene parent pom – this way it applies to all artifacts rather then just the lucene core.jar

The output manifest now looks like:

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven Bundle Plugin
Built-By: ryan
Build-Jdk: 1.6.0_13
Implementation-Vendor-Id: org.apache.lucene
Extension-Name: org.apache.lucene
Implementation-Title: org.apache.lucene
Implementation-Vendor: The Apache Software Foundation
Implementation-Version: 4.0-SNAPSHOT 1130132 - ryan - 2011-06-01 09:12
 :35
Specification-Title: Lucene Core
Specification-Vendor: The Apache Software Foundation
Specification-Version: 4.0.0.2011.06.01.09.12.35
X-Compile-Source-JDK: 1.5
X-Compile-Target-JDK: 1.5
Export-Package: org.apache.lucene.analysis;uses:="org.apache.lucene.ut
 il,org.apache.lucene.store,org.apache.lucene.document,org.apache.luce
 ne.analysis.tokenattributes,org.apache.lucene.index",org.apache.lucen
 e.analysis.tokenattributes;uses:="org.apache.lucene.util,org.apache.l
 ucene.index",org.apache.lucene.document;uses:="org.apache.lucene.util
 ,org.apache.lucene.analysis",org.apache.lucene.index;uses:="org.apach
 e.lucene.search,org.apache.lucene.util,org.apache.lucene.store,org.ap
 ache.lucene.document,org.apache.lucene.index.codecs,org.apache.lucene
 .analysis.tokenattributes,org.apache.lucene.analysis",org.apache.luce
 ne.index.codecs;uses:="org.apache.lucene.index,org.apache.lucene.util
 ,org.apache.lucene.store,org.apache.lucene.index.codecs.standard,org.
 apache.lucene.index.codecs.pulsing,org.apache.lucene.index.codecs.sim
 pletext,org.apache.lucene.index.codecs.preflex,org.apache.lucene.util
 .packed,org.apache.lucene.util.fst",org.apache.lucene.index.codecs.in
 tblock;uses:="org.apache.lucene.store,org.apache.lucene.index.codecs.
 sep,org.apache.lucene.util",org.apache.lucene.index.codecs.preflex;us
 es:="org.apache.lucene.store,org.apache.lucene.index.codecs,org.apach
 e.lucene.index,org.apache.lucene.util,org.apache.lucene.index.codecs.
 standard",org.apache.lucene.index.codecs.pulsing;uses:="org.apache.lu
 cene.index.codecs.standard,org.apache.lucene.util,org.apache.lucene.s
 tore,org.apache.lucene.index.codecs,org.apache.lucene.index",org.apac
 he.lucene.index.codecs.sep;uses:="org.apache.lucene.store,org.apache.
 lucene.util,org.apache.lucene.index,org.apache.lucene.index.codecs",o
 rg.apache.lucene.index.codecs.simpletext;uses:="org.apache.lucene.sto
 re,org.apache.lucene.index.codecs,org.apache.lucene.index,org.apache.
 lucene.util,org.apache.lucene.util.fst",org.apache.lucene.index.codec
 s.standard;uses:="org.apache.lucene.store,org.apache.lucene.index.cod
 ecs,org.apache.lucene.index,org.apache.lucene.util",org.apache.lucene
 ,org.apache.lucene.messages,org.apache.lucene.queryParser;uses:="org.
 apache.lucene.util,org.apache.lucene.search,org.apache.lucene.analysi
 s,org.apache.lucene.analysis.tokenattributes,org.apache.lucene.docume
 nt,org.apache.lucene.index",org.apache.lucene.search;uses:="org.apach
 e.lucene.util,org.apache.lucene.util.automaton,org.apache.lucene.inde
 x,org.apache.lucene.util.packed,org.apache.lucene.search.cache,org.ap
 ache.lucene.store,org.apache.lucene.document,org.apache.lucene.analys
 is.tokenattributes,org.apache.lucene.analysis,org.apache.lucene.searc
 h.spans",org.apache.lucene.search.cache;uses:="org.apache.lucene.util
 ,org.apache.lucene.search,org.apache.lucene.index,org.apache.lucene.u
 til.packed",org.apache.lucene.search.function;uses:="org.apache.lucen
 e.search,org.apache.lucene.index,org.apache.lucene.util",org.apache.l
 ucene.search.payloads;uses:="org.apache.lucene.search,org.apache.luce
 ne.search.spans,org.apache.lucene.index,org.apache.lucene.util",org.a
 pache.lucene.search.spans;uses:="org.apache.lucene.util,org.apache.lu
 cene.search,org.apache.lucene.index",org.apache.lucene.store;uses:="o
 rg.apache.lucene.util",org.apache.lucene.util;uses:="org.apache.lucen
 e.store,org.apache.lucene.index,org.apache.lucene,org.apache.lucene.s
 earch",org.apache.lucene.util.automaton;uses:="org.apache.lucene.util
 ",org.apache.lucene.util.fst;uses:="org.apache.lucene.util,org.apache
 .lucene.store",org.apache.lucene.util.packed;uses:="org.apache.lucene
 .util,org.apache.lucene.store"
Tool: Bnd-1.15.0
Bundle-Name: Lucene Core
Bundle-Vendor: The Apache Software Foundation
Bundle-Version: 4.0.0.SNAPSHOT
Bnd-LastModified: 1306934011182
Bundle-ManifestVersion: 2
Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt
Bundle-Description: Apache Lucene Java Core
Bundle-SymbolicName: org.apache.lucene.core
Bundle-DocURL: http://www.apache.org/