using snapshot bundles by maven

[maggu2810]

Using Maven a bundle is identified by the triple group id, artifact id, version. For snapshots the version contains the post-fix -SNAPSHOT. For bundles a snapshot build contains the post-fix (normally, if I am not missing something ) .qualifier that is translated to a timestamp (YYYYMMDDHHMM) at build time.

I would like to create a repository with different snapshot versions of the same bundle (in reality more than just one specific bundle, but let's keep it simple, to explain the problem at all).

Example:

• 1.2.3.201504291200
• 1.2.3.201504291230
• 1.2.3.201504291300

I get in trouble to fetch the different bundles using maven, as for maven the version is equal for all three bundles (1.2.3-SNAPSHOT).

You already implemented the pom extraction of a jar file and using this one to be able to fetch the bundle by GAV information.

Do you think it would be possible to create an aspect that replace the version in the pom file with the version of the bundle (independ of pom extraction), so a OSGi bundle override the version in the pom.xml with that one in the manifest?

[cornzy]

I think that is a conceptual problem of maven/OSGi/tycho.

In maven you cannot have multiple snapshot versions at the same time (as far as I know). But if you could distinguish the versions by other criteria maven gives you the ability to specify a version more precisely, for example "1.2.3.xyz-SNAPSHOT". OSGi on the other hand does only support version numbers in the form "n.n.n.qualifier". And it expects every version for a specific bundle to be globally unique! (Thats why the timestamp is used for every build).

You could specify how tycho should bild the qualifier of the OSGi version (by pattern) but unfortunately you could not change your maven version in a tycho build.

I don't think it would be the right way to change the maven versions using the OSGi version.

[maggu2810]

I don't think it would be the right way to change the maven versions using the OSGi version.

You are right, but if I cannot change the bundles (before building) myself (not my projects, have to do it before every build and drop the changes to merge upstream stuff later, ...) and would like to host multiple snapshots on a Maven repo, I have to solve it some way.

I could change the pom file in the jar to set the version to that one in the manifest (as explained above) and using the pom extractor. If there would be a configure option that this could be done transparent by package-drone it would be nice (but surely not a must).

[cornzy]

Maybe you could clarify your usecase a little bit. Are the bundles build by tycho? And you want to use these bundles as pure maven? I think I misunderstood because that seems to do not work at all. Hence where comes the bundles from and where do they go :)

[maggu2810]

I am using Karaf and want to install bundles using a maven repository.

To install a bundle I used

bundle:install mvn:groupid/artifactid/1.2.3-SNAPSHOT

It seems that I could also use the snapshot version instead of the version to define the maven dependency.

Namespace	Key	Value
mvn	snapshotVersion	1.2.3-20150429.132657-1
mvn	version	1.2.3-SNAPSHOT

So, using this one will solve it for me:

bundle:install mvn:groupid/artifactid/1.2.3-20150429.132657-1

Have to check if this is working for multiple 1.2.3-SNAPSHOTS with different snapshot versions in one repo.

[maggu2810]

Just to clarify the stuff a little bit more.

• I want to install bundles in a Karaf container (using a maven repo).
• So I have to create a maven repo, that contains that bundles.
• I have two different snapshot builds of a bundle
• The first Karaf container should use the snapshot 1 (1.2.3-SNAPSHOT / 1.2.3.201504291200)
• The second Karaf container should use the snapshot 2 (1.2.3-SNAPSHOT / 1.2.3.201504291230)

[maggu2810]

Ok, it is working. Sorry for interruption.

karaf@root()> bundle:update 180 mvn:org.eclipse.smarthome.io/org.eclipse.smarthome.io.transport.mqtt/0.8.0-20150429.132657-1
karaf@root()> bundle:list org.eclipse.smarthome.io.transport.mqtt
START LEVEL 39 , List Threshold: 50
 ID | State     | Lvl | Version            | Name                                   
------------------------------------------------------------------------------------
180 | Installed |  80 | 0.8.0.201504291326 | Eclipse SmartHome MQTT Transport Bundle

karaf@root()> bundle:update 180 mvn:org.eclipse.smarthome.io/org.eclipse.smarthome.io.transport.mqtt/0.8.0-20150429.142402-1
karaf@root()> bundle:list org.eclipse.smarthome.io.transport.mqtt
START LEVEL 39 , List Threshold: 50
 ID | State     | Lvl | Version            | Name                                   
------------------------------------------------------------------------------------
180 | Installed |  80 | 0.8.0.201504291423 | Eclipse SmartHome MQTT Transport Bundle

[ctron]

I truly understand your issue. There are a few additional bits of information.

Maven does have the qualified SNAPSHOT version (as you already mentioned) and I guess it should be possible to use this as well. At least the Unzip adapter does allow to use it.

Maven Tycho (and Eclipse PDE) allows you to replace ".qualifier" with some value during the build. This can be, but most not be, the a timestamp of the build. Maven Tycho allows you to use the timestamp of the last checkin (with git). So you do get the same timestamp between different builds if you don't make changes to the git repository in this directory. I am not sure if this is wise, but it is a way to do things.

[ctron]

I just read you comment while writing mine :wink: Sounds good! :smile:

[maggu2810]

Just to add a note again: Building a project that generates a lot of bundles using Maven Tycho, I realized, that the same value is used for all ".qualifier" (perhaps a configuration option). But the timestamp of the maven snapshot version differs for every bundle.

[ctron]

I guess a problem that would create the replacement of versions would be the fact this would break references between maven artifacts.

So assume A references B. It would do so by referencing B-SNAPSHOT, however only B.123 would exist. And this would cause an issue for Maven, I think?!

[ctron]

Yes, maven does generate a new timestamp. Not sure when. Tycho uses one for the whole build reactor (see https://wiki.eclipse.org/Tycho/Reproducible_Version_Qualifiers).

For maven there is maven.build.timestamp and maven.build.timestamp.format. Which should be stable for the whole build, the time the build was started.

But I am not sure if this is used for the uploaded SNAPSHOT version.

[maggu2810]

So, perhaps the feature I would like to have is that the tycho qualifier could be used to fetch maven artifacts.

It seems using the snapshotVersion to fetch the artifacts is working, so fit the snapshotVersion to the bundle version is perhaps something that could work.

But perhaps we should "know" what maven is using for uploaded SNAPSHOT version.

[ctron]

Just as additional input. For Eclipse SCADA we created a small tool "p2tom2", for converting a P2 repository to a Maven repository. This tool creates POM files from the metadata information of P2. It is a bit tricky to use, but works. All dependencies between OSGi bundles get re-created and added to the generated POM. In this case the Maven version really is the OSGi version. No matter what it would have been before.

The drawback is that this can only resolve "local" dependencies correctly. Externals (like slf4j, …) need to get provided in a properties file. Since OSGi bundles can reference to a Java package, which cannot be resolved against a plain Maven repository.

The long term goal is to let Package Drone do this job. :wink:

But for the version part, this creates a full Maven repository, which only has qualified versions, based on the OSGi versions. And no more "-SNAPSHOT" versions.

[ctron]

I am just thinking if it would be possible to create virtual maven artifacts (in addition) which the OSGi version. So you could have both.

[maggu2810]

That would be great. Perhaps we can using something similar to the -source artifacts.

For a OSGi bundle with Maven metadata we could create a virtual maven artifact that group id is changed to start with "pd.virt.osgi" or something similar, kept the artifact id and use the bundle version for the maven version. If the Maven metadata is missing, we could use "pd.virt.osgi" for the group id, the BSN for the artifact id and the bundle version for the maven version.

I think to create a additional virtual artifact by some intelligent aspect would be nice.

[maggu2810]

Or we could drop the maven artifact id (if available) and using always

• group id: pd.virt.osgi (example)
• artifact id: BSN
• version: bundle version

[ctron]

Well we could simply add a group ID to the provided channel metadata. Make it editable and let the user put it in. With a reasonable default as you suggested.

[maggu2810]

I reconsider our idea last night and find it an very useful extension. Let's define (as you stated above) a group id somewhere (e.g. channel metadata). A (new) channel aspect will use this group id (if present; if not present it is using a default group id or does nothing), the BSN and the bundle version to create a virtual artifact (pom.xml) that could be used by maven to fetch the bundle.

Later, if this is stable, we could decide if we want to extend the aspect to resolve import-package by using the export package information of all other bundles in the channel and create also dependency stuff in the pom.xml.

[ctron]

I think this sounds like a good plan!

[maggu2810]

How should I start? Reading the code of the maven extractor aspect to learn how to create a new aspect?

[ctron]

I guess it would really be best to create a new bundle and definitely to add a new aspect. Checking the code of the maven aspect might be a good idea. Or you can check the "Hasher" aspect (HashAspectFactory), which is even simpler. I am glad you volunteer :wink: and I will definitely help you!

[maggu2810]

But I am not sure if this is used for the uploaded SNAPSHOT version.

The deploy goal generates a unique version on deploy.

[maggu2810]

Some questions:

• Best practice for Package Drone to create a new bundle
• How to start with a simple channel aspect stub
• How could I iterate over all bundles (so OSGi bundles only) of a channel
• How could I fetch the OSGi information of a bundle
• How could I create a virtual pom file

[ctron]

A good starting point to all of this could be the "Hasher" aspect: HashAspectFactory in the bundle de.dentrassi.pm.aspect.common.

• Best practice for Package Drone to create a new bundle

Based on having a working Eclipse PDE setup: Create a new OSGi bundle for the Aspect itself. Create an additional one for a possible Web UI. The bundle should only be based on OSGi, not Eclipse UI or Equinox.

• How to start with a simple channel aspect stub

An aspect is an implementation of ChannelAspectFactory registered as an OSGi service. The easiest way to do this is to create a OSGi service as DS. Create an implementation of ChannelAspectFactory, create a new "Service Component" file in OSGI-INF and set at least the properties:

drone.aspect.id=<id of aspect>
drone.aspect.name=<label of aspect>
drone.aspect.version=1.0.0

The version is the version of the metadata. If the meta data format of this aspect changes (e.g. extracted data is used in the UI and the extracted format changes), package drone will recognized the change and offer solutions for re-generating the meta data.

Additionally an HTML file and/or plain text for a short description to the aspect:

drone.aspect.description.file=/OSGI-INF/help.html
service.description=Plain text help

Aspects can be assigned to a group (for grouping them in the UI):

drone.aspect.group.id=<group id>

The group has to be defined somewhere else by registering a service of de.dentrassi.pm.aspect.group.Group.

• How could I iterate over all bundles (so OSGi bundles only) of a channel

Actively iterating over a bundle is possible in general, but not for aspects. Aspects operate at a different layer. There is an API to access package drone from "external" and one to work from "internal". The latter one (internal) is called aspect. Therefore aspects are much more defined what the can do, but on the other side this allows to perform operations with the lock manager, the transaction manager and in a more defined context. For example if a channel aspect fails, all changes are rolled back atomically.

The ChannelAspect interface defines which aspect operations there are. Each method may return an instance for the specific operation. e.g. an Extractor if the aspect wishes to take part in the meta data extraction process.

When uploading an artifact the aspect operation is like:

1) Call all listeners with "pre" - this may veto the creation 2) Extract the meta data from the artifact BLOB – Extractor 3) Ask generated artifacts if they want to be regenerated 4) Call all virtualizers for this artifact – Virtualizer -> may trigger 1) to 4) again 5) Regenerate all generated artifacts which requested regeneration -> may trigger 1) to 5) again 6) Aggeragate the channel

• How could I fetch the OSGi information of a bundle

The "OSGi" aspect will extract most OSGi information. And provides an API to parse this from the meta data:

import de.dentrassi.pm.osgi.bundle.*;

BundleInformation bi = OsgiAspectFactory.fetchBundleInformation ( md );

• How could I create a virtual pom file

Implementing the channel aspect and providing an instance to Virtualizer.

[maggu2810]

Thanks for all that information.

[maggu2810]

I think we could close this one, as mvnosgi is implemented. Correct?

[ctron]

It was your bug initially ;-) If you like, just do so.