Migration to a Maven + Bnd based build system

[maggu2810]

We are in the process of changing the build system. We are not accepting new pull requests that are still using the old way from here on.

For further information see: Migration to a Maven + Bnd based build system

[maggu2810]

Migration to a Maven + Bnd based build system

The build system has been migrated to a Mave + Bnd based build system. We are not accepting new pull requests that are still using the old way from here on.

Background

openHAB is a java project that is build around the OSGi framework. openHAB add-ons are OSGi bundles.

Over the years the bnd software ("bnd is the engine behind a number of popular software development tools that support OSGi") has established itself as the defacto standard for creating OSGi bundles.

The openHAB maintainers have decided to move from the very Eclipse centric buildsystem to maven/bnd. A combination that not only allows to develop bindings seamlessly in other IDEs than Eclipse, but Java beginners and intermediates find a commonly known maven buildsystem.

Advantages of the new buildsystem

• Builds faster
• Offline mode builds
• Less buildsystem file overhead
• External dependencies per extension via the standard maven dependency system
• Unit tests are now part of the same project (Eclipse allows to create a unit test for a class via GUI now)

How to migrate

The easiest way to migrate is to use the create_openhab_binding_skeleton script in the bundles folder and create the basis for your migrated binding and than copy your binding into the new generated folder:

1. Generate your binding with create_openhab_binding_skeleton <binding name> "<your full name>" <your github name>. (This script also updates the CODEOWNERS and bom addons pom. So if you have already updated those files you need to clean that up).
2. Move your source from your src/main to the new binding folder in binding in bundles.
3. Move your README.md to the new binding folder. If you have any non standard additional files. Move them also (like images used in your readme)
4. Add maven dependencies to the pom.xml file if your add-on requires dependencies like so:

   <dependencies>
     <dependency>
       <groupId>org.apache.httpcomponents</groupId>
       <artifactId>httpclient-osgi</artifactId>
       <version>4.5.2</version>
       <scope>provided</scope>
     </dependency>
   </dependencies>

   Remove those included jar files. ONLY IF you can't find a maven central bundle you can keep the jar file in the lib/ directory.

5. Move ESH-INF to src/main/resources/ESH-INF
6. Move unit tests from a potential separate .test project into src/test.
7. Move integration tests into a new bundle, see the itests/ directory for examples.
8. Adapt src/main/feature/feature.xml to add any external dependencies, you must list them here, too. An example:

     <feature name="openhab-binding-lgwebos" description="LG webOS Binding" version="${project.version}">
          ...
         <bundle dependency="true">mvn:org.apache.httpcomponents/httpclient-osgi/4.5.2</bundle>
         <bundle start-level="80">mvn:org.openhab.binding/org.openhab.binding.lgwebos/${project.version}</bundle>
     </feature>

   There are two types of external libraries. Those that are ready for OSGi (like the one above) and those that aren't. The latter ones are listed with a wrap prefix keyword like this:

   <feature prerequisite="true">wrap</feature>
   <bundle dependency="true">wrap:mvn:io.netty/netty-common/4.1.34.Final$Bundle-Name=Netty%20Common&amp;Bundle-SymbolicName=io.netty.netty-common&amp;Bundle-Version=4.1.34</bundle>

   Please ask for details and review if in doubt.

Examples

• for an add-on with lib directory: https://github.com/openhab/openhab2-addons/tree/master/bundles/org.openhab.binding.allplay

How to build on the command line

Enter mvn clean install -DskipChecks in the project directory or itest project directory.

Subsequent calls can include the -o for offline.

For integration tests you might need to run: mvn clean install -DwithResolver.

Skip checks and tests if you just want to test the compilation process and perform the command from within the directory of the bundle that you want to build:

mvn clean install -DskipChecks -DskipTests -o

How to develop in the Eclipse IDE

1. Install Bndtools in your Eclipse IDE. (Is automatically installed if you have used the Eclipse Installer for openHAB development.)
2. Checkout the bnd based openHAB demo application: git clone --depth=1 https://github.com/maggu2810/openhab-demo.
3. Open the directory in Eclipse. Wait for the download and build to finish (about 3-5 minutes).
4. Checkout openhab2-addons: git clone --depth=1 https://github.com/openhab/openhab2-addons
5. In Eclipse File, Import, "Existing maven projects": Add your migrated project (not all projects!) to the workspace.
6. In Eclipse Package Explorer: Search for pom.xml in the demo-app project. Add your add-on as maven dependency like so:

   <project ...>
    ...
    <dependencies>
       <dependency>
           <groupId>org.openhab.addons.bundles</groupId>
           <artifactId>org.openhab.binding.astro</artifactId>
           <version>${project.version}</version>
           <scope>runtime</scope>
       </dependency>
    </dependencies>
   </project>

7. In Eclipse Package Explorer: Search for app.bndrun in the demo-app project. Double click (takes a few seconds).
8. Add your project to "Run requirements" via drag&drop from the package explorer. NOTE: If the binding you want to run/debug cannot be found in the Browse Repos list, or the list is empty, then follow these steps to rebuild the list of bindings in the BOM (Bill Of Material) and make your new binding visible from the IDE launch configuration:
   • using Git bash console go to git\openhab2-addons\bom\openhab-addons
   • recreate the BOM with: mvn -DskipChecks -DskipTests clean install
   • go back to Eclipse, close and re-open the app.bndrun file and now you should be able to find the binding in the list
9. Save and click "Resolve": Execute with "Run OSGi"

The demo application runs a slim set of openHAB core bundles including automations (next gen rules) and PaperUI (http://127.0.0.1:8080/paperui/). The startup should only take about 5 seconds and you are greeted by the openHAB console where you can type in console commands.

[wborn]

But it should be simple for Bintray.

It's sure a lot easier to create your own Bintray repo compared to uploading to Maven Central. Isn't it also easier for them being removed one day causing builds to fail? As fallback we'll probably still have such JAR artifacts in stable/milestone/snapshot releases.

But IMHO this would be just as problematic as when someone decides to delete their GitHub repo containing all the source code of a library. Since we don't store such sources or source JARs.

To mitigate these issues we could also setup an openHAB repo proxy to such 3rd party repos.

why are we trying to use local JAR files at all

I'd like to manage them using dependencies in POMs as well. It would be a good improvement to be able to update libraries and all their dependencies by just updating a version number. :-)

[maggu2810]

So, my offer: You (who) tells me an addon project name that also offers an OSGi based test project. I assume it should be one with no -- in the best case -- PR opened. Perhaps for the very beginning it should be one without a 3rd party JAR that is not available on a Maven repo.

I try to start the base work:

• split into one Maven + Tycho reactor and one Maven + Bnd reactor
• migrate the project to a Maven + Bnd one (as a direct child of bundles)
• migrate the OSGi based test project to a Maven + Bnd one (as a direct child of itests)

That's just an offer to help. If another one volunteers to do that job, I am happy, too.

[davidgraeff]

astro is a perfect match, isn't it? I would then migrate MQTT as soon as the remaining PRs are in, to test how well the migration process works.

[wborn]

I'm OK with Astro as well. The binding itself doesn't really matter as long as it's a bit representative of most projects. If we run into issues while migrating other add-ons we can always ask for more help. :wink:

[wborn]

Thanks for adding a migration guide @davidgraeff ! Didn't notice the first post was updated. :-)

[J-N-K]

Can we add that to the developer documentation (esp. the Part about eclipse)?

[J-N-K]

One thing regarding the miration guide: "remove '/lib'" is a good idea but will not work out-of-the-box for all bindings, since not all libs are availabe on bintray. And not all of those that are available can easily be used. So this should be checked before migration.

[davidgraeff]

@J-N-K There was a discussion about this between Markus and Wouter (I think?) and the agreement afaik is to move everything to (if necessary private) bintray/maven repos. For new contributions I also tend to say to either include the source code completely or only allow "established" libraries on maven central.

[J-N-K]

The end of this discussion was, that a decision by oh2-maintainers should be made about that. AFAIK this is still pending.

Including the source code maybe difficult. For example the Allplay binding includes a lib (Tchaikovsky) that is build with gradle and includes other libs that are not available on bintray (source is available on GitHub, though). Integrating that into our own build would greatly increase build time.

[davidgraeff]

My long time dream for openHab is Gradle as well, but Markus will slowly poke me to death with a spoon if I suggest that :D (But the speed advantage is real)

[J-N-K]

What I was trying to say: other project might use other build systems and integrating those with our own is quite some effort and may not be worth the time (compared to building them and uploading a binary to our own 3rd-party-libs repo).

[davidgraeff]

What I was trying to say: other project might use other build systems and integrating those with our own is quite some effort and may not be worth the time

The idea is to go to the maintainer of such a library and ask for a maven publish. There is even a maven plugin that uses github pages as a static repository site. We do not depend on their build system.

[lolodomo]

I am locked on the last step, that is launching the demo app. When I double-click on the file app.bndrun, it simply opens the file in the Eclipse editor. With what app should I open this file to start the demo app ?

[davidgraeff]

it simply opens the file in the Eclipse editor

It's like with pom.xml files: You can edit the source file or have the eclipse UI. The same for bndrun files: The default view might be text for you, but there should be the tab to show the UI (you see the glimpse of this selection in the screenshot above).

If not, your eclipse installation is missing bndtools (which is fatal). Please either manually install them or use the eclipse installer for openhab-core development. Let it do whatever it does, remove everything from the workspace again and then perform the steps from above.

[lolodomo]

Bndtools is now (manually) installed and I can see the dialog box with run requirements. But I can't add my binding from openhab2-addons. In the Browser repos at left, I see apparently only bundles from org.openhab.core.bom.openhab-core-index. No binding here, mainly core stuff.

If I click on the green +, it opens the "Add Bundle Requirement" dialog box but this is empty, I only see at left "Workspace default-ws" with nothing inside.

Same for Run Bundles. Project run path dialog box shows an empty workspace.

If I drag my project from the project explorer into the run requirements, I get this error: Error adding one or more requirements Error generating requirement Unable to derive identity from an object of type Project

[lolodomo]

I can go one step later, that is I can now add my binding and start OH. The problem is that it fails to start.

[cweitkamp]

Great tutorial. This will help others a lot. I walked through it step by step and want to add the following things:

5. Remove META-INF and OSGI-INF

We have to remove the build.properties file too.

7. Move unit tests from a potential separate .test project into src/main/test.

Move unit tests from a potential separate .testproject into src/test.

Additional changes need to be done in the files CODEOWNERS, the BOM (bom/openhab-addons/pom.xml, groupId) and features/karaf/openhab-addons/src/main/feature/feature.xml (bundle).

[lolodomo]

But I get a 404 error if I try to open the dashboard or Basic UI ! Is the demo app fully prepared/packaged to run UIs with the demo sitemap ?

[davidgraeff]

The demo only runs paperUI. You see that in the "Run requirements" section. There is only paperUI referenced. Add whatever you need there (you don't need to adapt pom.xml, because bnd knowns about all openhab-core and openhab-webui bundles)

Btw paperUI can be reached directly via http://openhab-ip/paperui without the dashboard (which is deprecated... next to PaperUI but that's another story.)

[lolodomo]

I have added some UI bundles. I don't see traces for Paper UI. Now when I try to access UI, I am requested a login/password for OSGi Management console ! I never had to enter that before. What has to be entered ?

[lolodomo]

Just to be clear, here is what I get by default (that is if I change nothing): No trace of Paper UI.

[lolodomo]

I tried admin / admin but then I still get 404 errors for any UI including Paper UI.

[davidgraeff]

Drag paperui over then from the "Browse Repos" window. It is called "org.openhab.ui.paper". No idea why credentials are required.

[lolodomo]

But I don't have org.openhab.ui.paper ! Are there several demo app ? I am using the one from openhab-core.

[davidgraeff]

Yes there are multiple one, that's why I added a screenshot to the list. There is only one org.openhab.demo.app bundle. The core has its own, but that one can only reference to core bundles and not to uis and is not really of any use. Markus wanted to remove it actually, but Kai not.

[lolodomo]

Ok I am not using the right one ! It looks like a new repo is requireed, openhab-demo.

[lolodomo]

Shall we still clone https://github.com/maggu2810/openhab-demo or is there now a openhab-demo repo ?

[davidgraeff]

That's the correct repo. Check your pom.xml of your demo app if it includes paperui:

    <dependency>
      <groupId>org.openhab.ui</groupId>
      <artifactId>org.openhab.ui.paper</artifactId>
      <version>${project.version}</version>
      <scope>runtime</scope>
    </dependency>

And what you can also do is to open the bndrun file and click the "Resolve" button, it will print some errors if something is not found etc.

[lolodomo]

It looks like the server is started. But still no WEB access. Doing a netstat command, I even have the feeling that there is no ports 8080 and 8443 opened ! I give up...

[davidgraeff]

The rest interface on /rest should work, it works for me. But yeah would be helpful if the others can comment on this, so that we get a better migration guide.

[lolodomo]

Like before, we should have a setup that by default allows starting and running the openHAB demo, including of course dashboard, Paper UI and Basic UI.

The constraint to update now a POM to add a particular binding is not as cool and easy as before (simply select your binding in Eclipse UI).

Making this easy and working again should be a priority IMHO. Without the ability to test your code in Eclipse, a lot of contributors will probably give up.

[davidgraeff]

@lolodomo The day is unfortunately only that long and almost everybody here does it in his freetime and the one guy that does it for a living (Markus) is in holiday.

The entire documentation for binding development need to be changed. And yes, changing a pom is not nice, there is a UI for this in eclipse as well though. I guess you can also drag&drop the project into that UI. We just need to find out what the easiest way is.

[davidgraeff]

CODEOWENERS will almost always give us merge conflicts. We should probably script migrate all extensions without tests and jars in one go?

[lolodomo]

So do we stop migrating our bindings individually ?

[davidgraeff]

@lolodomo No. The problem with the script migration is that some bindings should not yet be ported (those with open PRs). So please keep porting your bindings.

[lolodomo]

The RFXCOM binding is using an external lib, jd2xx. Source code is in sourceforge: https://sourceforge.net/projects/d2xx/files/ Last update is dated 2011 ! How do you want to handle such a dependency ? Can you accept keeping libraries in such a case ?

[J-N-K]

This is discussed e.g. in #5117 but not yet decided.

[davidgraeff]

I'd say we include the source files of that library (dual licensed and we pick the bsd one).

[lolodomo]

I discovered that the old way to start openHAB in Eclipse is still working well (and with all UIs available). The only problem is that the migrated bindings are no more presented. Don't know if this can be fixed. For the developers that need running and testing their bindings from inside Eclipse, until a clear and working guide is provided with the demo app, I would suggest to not migrate immediately.

[davidgraeff]

@lolodomo You are in the wrong topic for this suggestion. Here it is all about migrating all bindings as fast and clean as possible, because the core is currently "broken" until all bindings have been migrated.

After the migration and after the core has been fixed, we will think about how to develop bindings under these new conditions.

It may not even be Eclipse anymore, but Visual Studio Code as it is super easy to write addons for it. With an addon we can add "Run" and "Debug" buttons that do all the fancy OSGi container starting and maven dependency manipulation in the background.

[hakan42]

We should probably script migrate all extensions without tests and jars in one go?

I'd prefer one PR per binding. Else, you will have one humongous PR blocking everything else, and if it part of it needs to be reverted (e.g. some binding does not work even if it technically "migrated" as in "compiles"), you would break the complete master. Faced that pain too many times myself.

[paulianttila]

jd2xx nowadays cause more issues than it gains. Windows, linux and osx have native FTDI drivers, so binding is able to discover rfxcom bridge, but connection doesn’t work properly because of native driver. Additionally, most of the beginners try to introduce jd2xx bridge rather than direct serial port bridge, so I propose to remove whole bridge auto discovery and jd2xx staff from rfxcom binding.

[lolodomo]

@paulianttila this is a good solution to solve the problem :-) If we do that, this will break things at least for some users ?

[davidgraeff]

If we do that, this will break things at least for some users ?

Already configured things should not be influenced I guess? As far as I understood, jd2xx is only used for auto discovery. The readme should be amended to reflect that there is no discovery anymore.

[lolodomo]

No, my understanding of the code is that if a bridge was discovered with JD2xx then the connection to the serial port is done using JD2xx library. To avoid that, we will need at least the port name in the configuration but we don't have it.

[lolodomo]

@clinique : the netatmo binding could be a very difficult challenge with its 9 external libraries. You are probably the only one to understand/know how some of them were produced. What could be the solution for this binding ?

[clinique]

@lolodomo : my plan is to start by some simple bindings (Synop, OpenUV) to understand the move, and then go to Netatmo :) I did no see this thread before you pinged me, but I think it may answer question I raised on the forum right this morning !

[clinique]

It looks like the server is started. But still no WEB access. Doing a netstat command, I even have the feeling that there is no ports 8080 and 8443 opened ! I give up...

Hello @lolodomo , I'm stuck at the very same step. Did you find a solution ?

[davidgraeff]

@clinique Is the rest interface working? (/rest) You may need to add the "org.openhab.ui.dashboard" bundle to the xml and to the run requirements as well. This demo app just has never been really tested with uis.