Make lucene/solr a OSGI bundle through Ant [LUCENE-3167]

[asfimport]

We need to make a bundle thriugh Ant, so the binary can be published and no more need the download of the sources. Actually to get a OSGI bundle we need to use maven tools and build the sources. Here the reference for the creation of the OSGI bundle through Maven:

2421

Bndtools could be used inside Ant

---

Migrated from LUCENE-3167 by Luca Stancapiano, 12 votes, updated Sep 26 2013 Environment:

bndtools

Attachments: lucene_trunk.patch (versions: 2), LUCENE-3167_20130108.patch, LUCENE-3167.patch (versions: 3) Linked issues:

• SMX4-1306

[asfimport]

Bob Kerns (migrated from JIRA)

To respond to Robert Muir's question about why you (people who work on search engines) should do it.

I see two reasons:

1) People downstream may not KNOW how to do it, or have the time, or want to make all the possible choices over again, when people who know the engine and its packaging are more capable of sorting all that stuff out properly. I didn't get it right the first time (thank's to screwing up with) and now have to repeat. It's taken you guys a year and a half; obviously not trivial!

2) People downstream would like to do it in a way that does not outright conflict with other people downstream doing it! This is really a big, big deal. If I package it, and put it in a product, and someone else packages it, and puts it in their product – like for example, the Eclipse foundation – bad things can happen. I can't even predict WHAT bad things might happen, because I don't know what we might do differently. We could get lucky and not screw each other, but do I really want to trust the other guy – ALL the other guys – to do it right? Or do I do something like rename the packages to avoid any potential insanity? That has other problems...

Bottom line? I'd rather you guys do it – even if you don't do a good job of it. Even if I could do it 10x better, it is still 100x better for you to do it.

Maybe I'll be able to help later, once I've sorted out my immediate needs. But in the meantime, I just wanted to let you know the effort is appreciated – even if it doesn't turn out perfectly.

Thanks.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Bottom line? I'd rather you guys do it – even if you don't do a good job of it. Even if I could do it 10x better, it is still 100x better for you to do it.

Sure, its always better to ask someone else to take the burden. But I want the release process to continue to be smooth. The less unnecessary things we take on the better. The less manual stuff that needs to be checked the better.

We could package up linux RPMs and freebsd ports too, but we just leave this for downstream people. Could we do it better than those folks since we know the packaging? I don't know, but I don't really care either. Its better that we spend our time working on search engines so there are actual releases to put out there.

Recently I threw up a 4.2.0 release candidate on a wednesday night, by monday the release was out. By wednesday my friend forwards an email from freebsd showing they had updated their package already: http://svnweb.freebsd.org/ports?view=revision&revision=314108 . Thats pretty damn efficient downstream packaging and a relatively short release cycle. It required almost no manual effort: I'm insisting that we keep it this way.

So again, this is just like maven. I push back hard, really hard on the issue because I'm really just saying "don't add a feature without tests and force the release manager to manually verify the shit works". I said the same things about maven and someone (thanks Steve!) added automatic testing of maven and then I never said another word about it.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Just a very stupid question: Looking for "OSGI" in Wikipedia and other pages does not make it more clear to me. It confuses me more - this looks to me like completely useless and a duplicate of what Maven provides. Why do we need it when we release with Maven? Just because the Eclipse Foundation does not like Maven and wants to use their own "standard" OSGI? We decided to use Maven for publishing artifacts (in addition to the source and binary TGZ files). In my opinion, the Eclipse Foundation should provide a "converter" that makes Maven artifacts available to OSGI, they cannot force every project to support just another framework.

My last comment: If it is only adding some static strings to the META-INF of every JAR, who cares? But if you need crazy magic or must upload the JAR files to another site on the net, I am also -1!

[asfimport]

Tommaso Teofili (@tteofili) (migrated from JIRA)

My last comment: If it is only adding some static strings to the META-INF of every JAR, who cares? But if you need crazy magic or must upload the JAR files to another site on the net, I am also -1!

basically I do agree with the fact that we should keep the release process as smooth as possible, that's why I was not pushing too much this myself. I don't think we should upload artifacts anywhere else (and I don't think that's the point here), using the BND Ant tasks with our Ant based build is instead not really smooth, at least as far as I could do. My personal point here is if the Maven build already provides an "OSGi friendly" jar / manifest then (to me) we can mark this as resolved as I don't see why we should duplicate our efforts; probably we should instead fix the Maven configuration for the OSGi manifest as till now when I tried to deploy a released Lucene/Solr package into an OSGi container it didn't work correctly as the required manifest information wasn't there (which leads me to think that the problem may be related to the Maven profile used for the release).

My 2 cents.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

The official Lucene Maven artifacts as uploaded to Maven Central are not built by Maven! They are build by ANT. The POM files in dev-tools are a separate build environment thats not used for releasing.

This means the missing meta-inf must be added by ANT. The question I have: Why no simple change the JAR task and add the few additional key-value pairs? This would be a simple patch, if it is just some more-or-less static properties derived from some ANT build properties (like module name, version,...).

[asfimport]

Tommaso Teofili (@tteofili) (migrated from JIRA)

The official Lucene Maven artifacts as uploaded to Maven Central are not built by Maven! They are build by ANT

that's what I knew, but I then misunderstood your sentence when you said "we release with Maven", sorry for the confusion :)

This means the missing meta-inf must be added by ANT. The question I have: Why no simple change the JAR task and add the few additional key-value pairs? This would be a simple patch, if it is just some more-or-less static properties derived from some ANT build properties (like module name, version,...).

good point

[asfimport]

Nicolas Lalevée (migrated from JIRA)

If I simplify, Maven dependency management is about declaring that a jar depends on another. OSGi is more powerful/complex than that. It can do that jar to jar dependency, but it is not recommended by the advocates of OSGi. The prefered way is when dependencies are declared between the java packages. So instead of declaring a dependency on a jar, you declare a dependency on a java package. It is then like declaring a dependency on an API rather than an implementation. And then comes versioning into the mix : you have a version of an API (the java packages) and the version of the implementation (the jar/bundle). So that static metadata in the manifest is not trivial to maintain. Then here comes some tools to help, the patches here use "bnd". The java packages are somewhat part of the Java type system, so tools like bnd you can infer nearly automatically these dependencies.

Also note that contrary to the Maven world, the OSGi world doesn't have a world class reference repository. So you do not need to upload it anywhere else. I guess this issue is "just" about making the Lucene jars droppable in the OSGi environement as is.

Maybe if the Lucene committers doesn't feel enough confortable with OSGi, they should let it to some external OSGi packagers, just like there are debian packagers. Here is a exemple of an Ivy repository maintained by 'packagers': https://code.google.com/p/ivyroundup/. There was a tentative for OSGi but it stalled: https://github.com/glyn/bundlerepo

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Maybe if the Lucene committers doesn't feel enough confortable with OSGi, they should let it to some external OSGi packagers, just like there are debian packagers. Here is a exemple of an Ivy repository maintained by 'packagers': https://code\.google\.com/p/ivyroundup/. There was a tentative for OSGi but it stalled: https://github\.com/glyn/bundlerepo

Yes, I think someone downstream should do it, outside of this project.

I said the same thing about maven, that one didnt work out. But this had to do more with maven advocates lying about the necessity of it taking place in this project: turned out later this wasn't true and pretty much anybody can release anybody else's shit on maven central.

I won't make the same mistake twice.

[asfimport]

Tommaso Teofili (@tteofili) (migrated from JIRA)

reading again the whole history, and especially Robert's comment on eventually leveraging Ivy for this, I found https://ant.apache.org/ivy/history/trunk/osgi.html

I don't know if and how it could be useful, I'll have a look and in case let you know.

[asfimport]

Tommaso Teofili (@tteofili) (migrated from JIRA)

from a first inspection it doesn't seem to enhance the basic Ant + bndTools based approach too much therefore I'm not sure it's worth going that path.

[asfimport]

Nicolas Lalevée (migrated from JIRA)

As wrote earlier, Ivy is indeed useless to build an OSGi Manifest, it is "just" able to understand an OSGi remote repository. Here Ant + bndTools is the perfect tool combination for this. (BTW I'm a maintainer of Ivy).