Open asfimport opened 13 years ago
Luca Stancapiano (migrated from JIRA)
A starting point is here:
Downloading the biz.aQute.bnd.jar package from http://dl.dropbox.com/u/2590603/bnd/biz.aQute.bnd.jar and put it in the libs home of ant.
Then add this:
<bnd
classpath="src"
eclipse="false"
failok="false"
exceptions="true"
files="${common.dir}/lucene.bnd" />
in the lucene_trunk/lucene/common-build.xml file inside:
<macrodef name="build-manifest" description="Builds a manifest file"> <attribute name="title" default="Lucene Search Engine: ${ant.project.name}" /> <sequential> .....
and in the same directory create a lucene.bnd file containing:
Export-Package: *;-split-package:=merge-first
Need testing
Luca Stancapiano (migrated from JIRA)
Here a updated version using the correct classpath:
<property name="bndclasspath" refid="classpath"/>
<taskdef resource="aQute/bnd/ant/taskdef.properties" />
<bnd
classpath="${bndclasspath}"
eclipse="false"
failok="false"
exceptions="true"
files="${common.dir}/lucene.bnd" />
The ant classpath is different by the maven classpath so there are differences in the resulting 'Export-Package' variable in the MANIFEST.MF but both are ok
Luca Stancapiano (migrated from JIRA)
Some OSGI informations are inside the pom.xml . So if we want an automatism to create the OSGI attributes in the MANIFEST.MF, we have to read inside the pom.xml.template files.
Theese are the OSGI informations to add in the MANIFEST:
Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt (project.licenses.license.url in the parent pom.xml.template)
Bundle-SymbolicName: org.apache.lucene.misc (project.groupId+project.artifactId in the pom.xml.template)
Bundle-Name: Lucene Miscellaneous (project.name attribute in the pom.xml.template)
Bundle-Vendor: The Apache Software Foundation (from the parent pom.xml.template)
Bundle-Version: 4.0-SNAPSHOT ($version variable from ant)
Bundle-Description: Miscellaneous Lucene extensions (project.description from pom.xml.template)
Bundle-DocURL: http://www.apache.org/ (project.documentation.url in the parent pom.xml.template)
Else we should duplicate the informations. What is the better road?
Gunnar Wagenknecht (migrated from JIRA)
The approach I implemented in my patch (attached to LUCENE-1344) used template files for BND. I wonder if both - Ant as well as Maven could use those files.
Luca Stancapiano (migrated from JIRA)
It depends by the structure of your templates. Can you tell me how is organized the tree of the template files?
Luca Stancapiano (migrated from JIRA)
I expose this patch:
<macrodef name="build-manifest" description="Builds a manifest file"> <attribute name="title" default="Lucene Search Engine: ${ant.project.name}" /> <attribute name="bndtempDir" default="${build.dir}/temp"/> <sequential> <xmlproperty file="${ant.file}" collapseAttributes="true" prefix="bnd"/> <property name="bndclasspath" refid="classpath"/> <taskdef resource="aQute/bnd/ant/taskdef.properties" /> <mkdir dir="@{bndtempDir}"/> <bnd classpath="${bndclasspath}" eclipse="false" failok="false" exceptions="true" files="${common.dir}/lucene.bnd" output="@{bndtempDir}/${final.name}-temp.jar" /> <copy todir="${common.dir}/build" flatten="true"> <resources> <url url="jar:file://@{bndtempDir}/${final.name}-temp.jar!/META-INF/MANIFEST.MF"/> </resources> </copy> </sequential> </macrodef>
It rewrites the build-manifest macrodef because bndlib cannot append the information of the manifest.mf. I moved the information of the manifest in the lucene.bnd file appending the new osgi info:
Export-Package: *;-split-package:=merge-first Specification-Title: Lucene Search Engine: ${ant.project.name} Specification-Version: ${spec.version} Specification-Vendor: The Apache Software Foundation Implementation-Title: org.apache.lucene Implementation-Version: ${version} ${svnversion} - ${DSTAMP} ${TSTAMP} Implementation-Vendor: The Apache Software Foundation X-Compile-Source-JDK: ${javac.source} X-Compile-Target-JDK: ${javac.target} Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt Bundle-SymbolicName: org.apache.lucene.${name} Bundle-Name: Lucene Search Engine: ${ant.project.name} Bundle-Vendor: The Apache Software Foundation Bundle-Version: ${version} Bundle-Description: ${bnd.project.description} Bundle-DocUR: http://www.apache.org/
I tested on lucene and solr modules and all jars are created with the correct OSGI info in the manifest.mf. Unluckily bndlib is not flexible so if you use bndlib you are forced to:
Luca Stancapiano (migrated from JIRA)
I created an issue on https://github.com/bnd/bnd/issues/70 to parametrize the bnd ant task
Luca Stancapiano (migrated from JIRA)
I send a new patch.
Steven Rowe (@sarowe) (migrated from JIRA)
Hi Luca,
I downloaded the bnd jar and applied your patch, then ran ant jar-core
under lucene/
. This produced a jar, and didn't seem too awfully slow. The generated MANIFEST.MF
looks mostly okay - see below for some issues.
Can you do some timings with and without the OSGI stuff for creating jars? If it doesn't take much extra time to produce the OSGi-compatible manifests, I think it would be okay to always produce them.
Your patch doesn't have any special handling for Solr's war file manifest - does OSGi care about war files? (I know nothing about OSGi.)
The bnd jar is Apache licensed, so we can put it into our repository, rather than require people to download the jar. You can see an example of this in lucene/build.xml
and lucene/common-build.xml
with the Maven Ant Tasks jar - take a look at the definitions of the generate-maven-artifacts
target and the maven-ant-tasks.classpath
path.
Some stuff that needs to be addressed before this can be committed:
LUCENE-3167.patch
. See http://wiki.apache.org/lucene-java/HowToContribute#Creating_a_patch for more info.jarify
macro invokes the build-manifest
macro. The jarify
macro may need more parameters to be able to handle both Solr and Lucene needs.MANIFEST.MF
, the Bundle-Description
value is not interpolated properly - here's what I get: Bundle-Description: $\{bnd.project.description\
}MANIFEST.MF
, DSTAMP
, TSTAMP
, and TODAY
each have entries, but they don't appear in the lucene.bnd
template - if possible, these should be eliminated.lucene.bnd
, you have excluded $\{user.name\
} from the Implementation-Version
value - please re-sync with the unpatched value.lucene.bnd
, should Bundle-DocUR
be Bundle-DocURL
?Luca Stancapiano (migrated from JIRA)
Hi Steven,
I send a new updated patch.
I added two new stamp properties in the build-manifest macro (start.touch.time and end.touch.time) that log the milliseconds of the process.
War files in OSGI are worked as the jar files. If the OSGI repository has functionalities to work with containers, it takes the informations directly by the bundle. The MANIFEST.MF file doesn't include informations about containers.
I added the bnd library from http://dl.dropbox.com/u/2590603/bnd/biz.aQute.bndlib.jar (actually in the dropbox there is the only version for ant. See: http://www.aqute.biz/Bnd/Download) and added it to the ant classpath how for the 'generate-maven-artifacts' target.
Here the responses to the tasks:
1 - checked the box to grant the Apache license.
2 - Renamed the patch according the convetion.
3 - Deleted the bnd configuration for solr. Now only the build-manifest macro declared in the common-build.xml of lucene project is used. But I was forced to declare the attributes @{title} and @{implementation.title} as properties inside the build-manifest macro, else they didn't seen in the external file lucene.bnd.
4 - I see the correct value of ${bnd.project.description} because the property is created through the configuration : <xmlproperty file="${ant.file}" collapseAttributes="true" prefix="bnd"/> inside the build-manifest macro. Maybe I didn't added all in the previous patch. Let me know if the problem persists.
5 - I excluded the DSTAMP, TSTAMP, and TODAY properties by the bnd configuration through the property: -removeheaders . The main problem is that the bnd ant task takes all the ant properties starting with an uppercased lecter and add them without ask. Should be a bnd property -inherit (true/false) that tells if import the ant properties but it doesn't work. This problem is signed in: https://github.com/bnd/bnd/issues/72. An other important thing is that the 'Name' ant property declared in some build.xml is not accepted by the bnd ant task. In the bnd ant task code there is an hard exception if the 'Name' property is found:
if (header.equalsIgnoreCase("Name")) {
error("Your bnd file contains a header called 'Name'. This interferes with the manifest name section.");
continue;
}
So I was forced to rename the 'Name' property and its references in 'LuceneName'
6 - Added the ${user.name} property in the Implementation-Version manifest property
7 - Renamed the Bundle-DocUR property to Bundle-DocURL
Steven Rowe (@sarowe) (migrated from JIRA)
Hi Luca,
I'll take a look at your new patch today or tomorrow.
Have you done any timings yet?
I don't understand a couple of things you wrote:
War files in OSGI are worked as the jar files.
Do you mean that OSGI treats .war files the same as .jar files?
If the OSGI repository has functionalities to work with containers, it takes the informations directly by the bundle. The MANIFEST.MF file doesn't include informations about containers.
What is a container? What is a bundle? Why does it matter that MANIFEST.MF does not include information about containers? How are these things related to the other topics under discussion on this issue? (I wasn't kidding when I wrote that I know nothing about OSGi.)
Luca Stancapiano (migrated from JIRA)
Hi Stewen
I've done the timing adding the two tstamp properties, so if you start ant will see in the log something as:
Time starting for the build of manifest.mf: 10:55.50.186 ..... Time ending for the build of manifest.mf: 10:55.51.821
The time for each module changes from 1 to a mx of 5 seconds according the number of dependencies during the compilation.
About the second point you must not care about the wars because war and jar are read in the same manner in a OSGI repository. External utilities can add jobs to recognize Servlet, EJB or other javaee components but this work is delegated to the OSGI repository.
I've told about javaee containers and I'm sure you know them. A package (jar or war) containing the OSGI informations is called 'bundle'.
So, don't care about the war. They will be deployed without problems in a OSGI repository!!
Steven Rowe (@sarowe) (migrated from JIRA)
This patch fixes various problems with the created manifests.
I measured OSGI-compliant manifest creation time for all 38 non-example jars (and the Solr war), and the total cost was 153.5 seconds, which is about 4 seconds per bundle on average.
I also measured manifest creation time without the OSGI stuff, and it's basically instantaneous: single-digit milliseconds on average per jar/war.
Maybe 4 seconds per jar/war is okay? I'll wait a day or two before I commit so people can object or suggest alternatives.
One alternative to always building the OSGI manifests would be to only build them when preparing a release.
Steven Rowe (@sarowe) (migrated from JIRA)
Here's the correct patch.
Robert Muir (@rmuir) (migrated from JIRA)
I measured OSGI-compliant manifest creation time for all 38 non-example jars (and the Solr war), and the total cost was 153.5 seconds, which is about 4 seconds per bundle on average.
Ouch: any idea why it takes so long? I think adding 153 seconds to the compilation time is very painful for developers.
FYI: I had encoding problems with the patch (its ISO-8859-1 not UTF-8?)
Steven Rowe (@sarowe) (migrated from JIRA)
Ouch: any idea why it takes so long? I think adding 153 seconds to the compilation time is very painful for developers.
AFAICT, the bnd tool visits everything in the classpath and records everything it finds in the manifest. Luca, maybe you can add some more info here?
FYI: I had encoding problems with the patch (its ISO-8859-1 not UTF-8?)
How did you experience these problems? I generated it using (Cygwin's) svn diff on Windows 7. I skimmed the patch and didn't see any above-ASCII chars. The patch applies without complaint using Cygwin's patch.
Robert Muir (@rmuir) (migrated from JIRA)
Steven and I got to the bottom of this: the issue was the ant license files in lucene/lib. These have a ® sign encoded in ISO-8859-1.
Luca Stancapiano (migrated from JIRA)
it takes time because the bndlib is tied to the compilation of the sources. bndlib navigates recursively in the classpath of each module. More the dependencies are lot, more time is used. It is important to create the list of all the dependent classes
Steven Rowe (@sarowe) (migrated from JIRA)
Because OSGi manifest creation is so slow, I will not commit this patch as-is.
I think a good compromise would be to rewrite the make-manifest target to only make OSGi-conformant manifests either when a system property has been set from the command line or when generating release artifacts. Or maybe only when generating release artifacts?
Luca Stancapiano (migrated from JIRA)
I imagine a property like 'dev' that excludes all heavy builds if it is not just ready. So it could be used for future ant tasks
Steven Rowe (@sarowe) (migrated from JIRA)
I imagine a property like 'dev' that excludes all heavy builds if it is not just ready. So it could be used for future ant tasks
I disagree. The name of the property should tell people what it's for. 'dev' is the opposite of that. Also, functionality like this should be individually configurable.
Luca Stancapiano (migrated from JIRA)
The risk for a property for a single functionality is that someone can forget it during the publishing. Surely I will add it but I think a global property (like 'release' or 'dev') that starts sub-tasks (osgi bundle in this case) too can be useful
Luca Stancapiano (migrated from JIRA)
I submit the new patch including the switch for the osgi compiler. I add a new property 'development'. This property could be used switch many other cases that can decrease the performances during the build. Here an example:
ant clean default
It will start in 'publish' mode. Actually only the osgi compile is added.
ant clean default -Ddevelopment=true
It will start in 'development' mode. The osgi compile will be removed and used the default manifest.mf
ant clean default -Ddevelopment=
Same thing as before
Steven Rowe (@sarowe) (migrated from JIRA)
Luca, you dropped the changes from my patch to solr/
and modules/
. Please put them back.
I add a new property 'development'. This property could be used switch many other cases that can decrease the performances during the build.
As I have stated previously, I don't like this idea.
The default build with no properties specified should be development mode (i.e., don't do the extra work needed to build OSGi manifests). The Lucene/Solr build is for the developers; it must be as fast as possible by default.
There should be a property named "build.osgi.manifests" or something similar that says what's happening, rather than hiding behind some anonymous "publish mode". That is, don't call the property "development" or "publish.mode". OSGi manifest building will be the only optional performance-decreasing element in the build, so there is no reason to generalize it at this point.
Luca Stancapiano (migrated from JIRA)
I tried it and it works and I like it. You are free to change it. Cheers
Steven Rowe (@sarowe) (migrated from JIRA)
Luca, you chose to implement your changes in a form that I will not commit - I think I was pretty clear about that. You too are free to change it. Leaving it to me means that it will be a while before it gets committed, since I have 10 other things I care about more. Cheers
Ryan Hill (migrated from JIRA)
Is [#comment-13113588] still the only problem with getting this committed?
Steven Rowe (@sarowe) (migrated from JIRA)
Is [#comment-13113588] still the only problem with getting this committed?
No. [#comment-13113617] and [#comment-13124118] mention other problems.
Steven Rowe (@sarowe) (migrated from JIRA)
The way forward, as I see it:
Ryan Hill (migrated from JIRA)
Thanks for the helpful digest. Your approach sounds completely reasonable to me.
I don't fiddle with ant much, but I'll take a look.
Robert Muir (@rmuir) (migrated from JIRA)
Out of curiousity, now that we are using ivy in our ant build, has anyone investigated the upcoming ivy support for OSGI?
Would it do what we want? http://ant.apache.org/ivy/history/trunk/osgi.html, http://ant.apache.org/ivy/history/trunk/osgi/standard-osgi.html, etc.
It seems to use the same utilities here, so I have a few concerns:
anyway I know nothing of OSGI, so maybe this is useless, I just wanted to bring it up.
Nicolas Lalevée (migrated from JIRA)
Ivy won't help you for this issue which is about adding OSGi metadata into the jar. Ivy can understand OSGi manifests but it is unable to write them. Tools like bnd (used in the patch IIRC) is definitively the way to go.
Jukka Zitting (migrated from JIRA)
Is anyone actively working on this (i.e. is Luca's patch from October 2011 the latest one)? If not, I can take a look at addressing the concerns raised above.
Michael McCandless (@mikemccand) (migrated from JIRA)
Hi Jukka, I don't think anyone is working on this (we don't really collectively have enough OSGI expertise around...). So if you can help out that would be awesome! Thanks.
Tommaso Teofili (@tteofili) (migrated from JIRA)
sure, and I'd be happy to lend a hand here.
Luca Stancapiano (migrated from JIRA)
Hi guys, Nicolas, I confirm for the October 2011... in that time the patch worked.....I'm wondered that none has still committed the work...I will be here for help. Let me know!
Jukka Zitting (migrated from JIRA)
We discussed this briefly with Tommaso and a few OSGi experts last week.
While adding bundle metadata to the Lucene jars is fairly straightforward to implement, doing so comes with a commitment to maintain correct semantic versioning at the package-level (see http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf). Without such a commitment the package version information will eventually become outdated and incorrect, which can lead to tricky problems for OSGi deployments. Most notably properly managed package versions will often not follow the main product version numbers, as even when backwards-incompatible changes is introduced in one package, another package might still remain unchanged.
The best incremental solution here could be to initially export only one or two key packages for which semantic version information can be maintained without too much overhead going forward. A good starting point could be something like a separate facade package for which stronger-than-usual backwards compatibility promises can be made. Once there's more OSGi experience in the core Lucene community, more packages can be exported to OSGi clients.
Robert Muir (@rmuir) (migrated from JIRA)
This seems like a significantly more complicated issue then than just modifying the ant build to add metadata. Maybe this build step should be addressed first (so someone can optionally do it with ant if they want), just like maven and leave the thornier back compat issue for another issue.
From this PDF it seems OSGI imposes its own backwards compatibility policy that conflicts with ours, and would be very difficult to maintain and detect if they were correct.
I don't see why this would be useful at all: in basically every release binary compatibility is usually broken in some way (and even if its not technically broken, maybe a methods behavior changes or something like). Different lucene modules depend on each other and are only tested with the same version: today its not tested to use lucene-foobar-3.2.jar with a lucene-core-3.1.jar.
So the versioning in that whitepaper doesn't make much sense: I think if we are going to put OSGI information in the packages we should simply bump the major version for every release: even bugfix releases.
Gunnar Wagenknecht (migrated from JIRA)
Jukka, these are all good point. I'm the maintainer of the Lucene OSGi bundles in Eclipse with quite a bit experience in OSGi-ifying libraries. There is a trade-off between satisfying all of the points and being pragmatic.
Having real semantic versioning at the package level would be good for OSGi. However, for the time being, I simply use the main Lucene version for all package exports. That works today and should also be a first step here.
Future work (like removing split packages and package level versioning) should be done as separate steps.
Robert Muir (@rmuir) (migrated from JIRA)
Wait: I'm saying I'm not sure these additional backwards compat things should be done at all.
Changing things about the backwards compatibility policy to make development and releasing more difficult is serious and I'm not seeing the benefit here.
A better approach would be to adapt the OSGI integration to work well with Lucene's existing backwards compatibility policy (http://wiki.apache.org/lucene-java/BackwardsCompatibility) instead of the other way around.
Steven Rowe (@sarowe) (migrated from JIRA)
I count 280 packages in Lucene/Solr.
Steven Rowe (@sarowe) (migrated from JIRA)
@rmuir wrote:
A better approach would be to adapt the OSGI integration to work well with Lucene's existing backwards compatibility policy (http://wiki\.apache\.org/lucene-java/BackwardsCompatibility) instead of the other way around.
What Gunnar Wagenknecht says he's doing now seems close?:
Having real semantic versioning at the package level would be good for OSGi. However, for the time being, I simply use the main Lucene version for all package exports. That works today and should also be a first step here.
Robert Muir (@rmuir) (migrated from JIRA)
This conflicts with that PDF though... I guess I'm saying I disagree with that PDF completely :)
Steven Rowe (@sarowe) (migrated from JIRA)
I disagree with the whitepaper too, except the part where is says that semantic versioning is optional.
Robert Muir (@rmuir) (migrated from JIRA)
Then we are back to my original suggestion above, that we start with just the ant integration as maven was done and don't try to change versioning nor back compat (this will be controversial, at least I will become really really annoying).
Tommaso Teofili (@tteofili) (migrated from JIRA)
In my opinion doing and (especially) maintaining the OSGi packages from ant would be a pain, on the other hand I can see that keeping two different versioning could be hard as well. That's the reason way I thought that, at least to start (as Jukka told), a facade package for explicitly exporting the main stuff for OSGi could be a solution. There someone with enough OSGi experience could take care of the whole thing without causing much impact on the project. What I personally look for is having something that works out without much pain and Jukka's proposal looks to me to go in that direction. Obviously if there are other ideas I think that they'd be more than welcome. Gunnar's proposal could be ok as a starting point as well, but actually I don't know how hard it'd be to do that automatically with our build mechanism.
Robert Muir (@rmuir) (migrated from JIRA)
What Gunnar Wagenknecht says he's doing now seems close?:
Right, which begs the question if people who understand this stuff can already do this downstream, why should we (people who work on search engines) do it.
Its just more stuff to go wrong when trying to release.
Tommaso Teofili (@tteofili) (migrated from JIRA)
now trying to rework Luca's patch to see if we can improve it, and have the OSGi manifest build disabled by default but maybe enabled for release and on demand (e.g. -Dosgi=true).
Luca Stancapiano (migrated from JIRA)
great, daje Tom ;)
Tommaso Teofili (@tteofili) (migrated from JIRA)
New not final version of the patch.
I'm not an Ant expert so I may be missing something, if that's the case feel free to correct / point me to where / how fix things.
With this new one it's possible to run ant compile -Dosgi=true to have OSGi ready jars' manifests so that single modules can be installed and started in an OSGi container. By default however the OSGi build doesn't happen.
What is still missing is a smart and non intrusive way of setting correctly the OSGi exported-package directive. Currently that's set to: org.apache.lucene*;version=${version},org.apache.lucene.*;version=${version};-split-package:=merge-first for all the bundles which is not correct.
As a side note I noticed that in lucene/core/build.xml the classpath is not set while is set to the core files in the root build.xml; this seems to me a bit odd, is there any specific reason for that? In the patch I temporarily added the correct classpath in lucene/core/build.xml (which is needed for the OSGi build for the core to export the packages).
We need to make a bundle thriugh Ant, so the binary can be published and no more need the download of the sources. Actually to get a OSGI bundle we need to use maven tools and build the sources. Here the reference for the creation of the OSGI bundle through Maven:
2421
Bndtools could be used inside Ant
Migrated from LUCENE-3167 by Luca Stancapiano, 12 votes, updated Sep 26 2013 Environment:
Attachments: lucene_trunk.patch (versions: 2), LUCENE-3167_20130108.patch, LUCENE-3167.patch (versions: 3) Linked issues: