Move debian package building to a separate repo

[kaikreuzer]

The issues on this repo are a wild mix of problems of the distro itself and the debian packaging. We have separate repos for Docker, Synology and possibly soon Snappy, so I would claim that it makes sense to move the debian package build as well to a separate repository.

As a side effect, this would also heavily decrease the size of our build jobs as discussed with @theoweiss a while ago.

@theoweiss Would you be ok with this and take the lead on it?

[theoweiss]

@kaikreuzer I'm ok with that, but it will take some time to get this done. Do you know a maven plugin to download a file (distro tar.gz) from a url?

[kaikreuzer]

Yes, see right here: https://github.com/openhab/openhab2-addons/blob/master/addons/voice/org.openhab.voice.marytts/pom.xml#L22

[theoweiss]

Looks good. How should we proceed?

[kaikreuzer]

Make a suggestion on the repo name and I'll create it and assign you as the maintainer. How about "openhab-debian"? Or better "openhab-apt"?

[theoweiss]

What do you think about openhab-common-linux (or openhab-ospackage) so we're free to add rpm packaging sometime? There is a gradle plugin to achieve this: https://github.com/nebula-plugins/gradle-ospackage-plugin, e.g. elasticsearch used it https://github.com/elastic/elasticsearch/blob/master/distribution/build.gradle.

[kaikreuzer]

Hm, neither common-linux nor ospackage seem to be common terms in the Linux community, so the name does not mean a lot for most people. Would "openhab-linuxpkg" maybe be an alternative?

[theoweiss]

Sounds good. I've thought about common-linux because Synology and Snappy are also linux packages.

[kaikreuzer]

Yes, and we have separate repos for them, so this was exactly the reason, why I didn't want to call it common-linux (the "common" sounds like judging what is normal and what isn't) - the "pkg" slightly refers to the packet management, what deb and rpm are mainly called.

[theoweiss]

Expressive names are always difficult to find :-( .

[kaikreuzer]

Yeah, one of the biggest problems in software engineering...

Repo is set up and you're the admin: https://github.com/openhab/openhab-linuxpkg

Have fun :-)

[theoweiss]

I thought of common in the sense of "well known" or "widespread".

[theoweiss]

Let's roll!

[theoweiss]

BTW your aware of this #302 ?

[kaikreuzer]

@theoweiss Any progress on this? I would really love to have a separation of concerns between these repos.

[theoweiss]

I've started some dev here: https://github.com/theoweiss/openhab-linuxpkg/tree/dev I'm struggling with some gradle/groovy stuff which is new to me. But I think it's a good way to go. I'm sure to get this done, but moreover the end of year race has begun for paid work and family stuff, therefore don't expect fast results. If there are any volunteers (with gradle experience) your very welcome to help out (may be @BClark09 ?).

[kaikreuzer]

Would be great to have @BClark09 joining the effort - I would want to wait too long; I plan a new beta release mid of December and it would be perfect to have the repos separated by then.

[BClark09]

I don't know much about gradle I'm afraid, but will help where I can. I've just sent a PR to the dev branch to update the paths.

[theoweiss]

I've just opened a PR https://github.com/openhab/openhab-linuxpkg/pull/1 which I'll merge the next days and an issue https://github.com/openhab/openhab-linuxpkg/issues/2 for discussing the further development. I invite everybody who's interested in deb packaging to join the discussion, especially @kaikreuzer, @BClark09 and @ThomDietrich.

[theoweiss]

Hi @kaikreuzer,

I suggest to add a new team called "LinuxPkg Maintainers" and want to envite @BClark09 to get a member of this team. If he accepts he should also become a member of the bintray openhab group https://bintray.com/openhab to be able to put releases to bintray.

Regards, Theo

[theoweiss]

BTW: I'm away from my email for the weekend and will be back on Monday evening.

[BClark09]

I'd love to help and appreciate your confidence in me although I've never done anything like that before and am only just learning the deb packaging ropes as it were. I'm also in my final year of my PhD so you may see me drop out of radar a couple of times. I would happily help maintain the LinuxPkg distro where I can though.

[kaikreuzer]

Many thanks for your work @theoweiss! I have created the @openhab/linuxpkg-maintainers group as requested. Please also note https://github.com/openhab/openhab-linuxpkg/issues/4.

[kaikreuzer]

@theoweiss & @BClark09 Happy New Year! Where are we regarding the move? Once we can remove the debian code from the distro itself, please also make sure to move/close all of [these issues] over to https://github.com/openhab/openhab-linuxpkg/issues. Thanks!

[BClark09]

Hi @kaikreuzer, Happy New Year to you too!

I have had a good deb package builds running on the openhab-linuxpkg branch, but I feel that there needs to be some sort of script that removes older nightly builds from Bintray (If I've understood correctly, we'll still use Bintray for them) before it's ready to be used. I'm assuming @theoweiss will know more about what's involved in that process and what is outstanding to make that happen.

[theoweiss]

The next steps are: cleanup and merge the two open PRs. Then integrate the linuxpkg project into the cloudbees build. I had planed to tests the jenkins integration with my local test server, but had to move the test till tomorrow evening.

[theoweiss]

Jenkins test worked like a charm :-)

At Bintray: https://bintray.com/theoweiss/apt-repo2/openhab2/2.0.0%7E20170103231411

@kaikreuzer I would add the project to cloudbees tomorrow. Do we have a bintray api key for the openHAB organisation? Otherwise I would use mine ;-(

[kaikreuzer]

@theoweiss Yes, we have! You will find it configured as BINTRAY_API_KEY and BINTRAY_USER at https://openhab.ci.cloudbees.com/configure.

[shoeper]

I'd like to mention that the current snapshot repository (https://openhab.ci.cloudbees.com/) is quite slow (< 200 KiB/s). Would it be possible to move the downloads to another (faster) mirror?

[kaikreuzer]

As mentioned by Theo above: It will be hosted on Bintray.

[shoeper]

Sorry, overread that. Good news :+1:

[theoweiss]

@kaikreuzer it seems like we've already used up the disk usage quota?

[kaikreuzer]

So we should first remove the debian package from the distro build - then we will use much less space!

[theoweiss]

Hmm, I hoped to get a smooth transition to the new apt snapshot repo, with both repos available.

[kaikreuzer]

I have reconfigured the distro build plan to only keep the artifacts of a single build (before of the last two builds), so I hope this frees up enough disk space for your new plan to set up!

[theoweiss]

I've just thought about a similar solution. Can we just manually delete some of the older builds?

[kaikreuzer]

These older ones don't take any space, because the artifacts are already removed. As I said, only for the very latest build, the artifacts are kept.

[kaikreuzer]

The "disk used" value is probably not updated yet. Just go ahead and create your build, it should work.

[theoweiss]

Ok.

[theoweiss]

Nice, it worked (,so far)!

@BClark09 could you please test the snapshot from the official openHAB repo. Thanks in advance.

[BClark09]

@theoweiss Will do as soon as I can! From what I noticed in the cloudbees log, calculateMetadata also needs to be run right?

[theoweiss]

Not necessarily, we will see.

[BClark09]

Input:
echo "deb https://dl.bintray.com/openhab/apt-repo2 unstable main" | sudo tee /etc/apt/sources.list.d/openhab.list
sudo apt-get update

Output: 
W: GPG error: https://dl.bintray.com unstable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 075721F6A224060A

Input:
gpg --keyserver pgpkeys.mit.edu --recv-key 075721F6A224060A
gpg -a --export 075721F6A224060A | sudo apt-key add -

Output:
gpg: key A224060A: public key "openHAB Bintray Repositories <owner@openhab.org>" imported
OK

Input:
sudo apt-get update && sudo apt-get upgrade

Output:
The following packages will be upgraded:
  openhab2-offline
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Get:1 https://dl.bintray.com/openhab/apt-repo2/ unstable/main openhab2-offline all 2.0.0~20170104233230-1 [186 MB]
Fetched 186 MB in 2min 9s (1,430 kB/s)
Unpacking openhab2-offline (2.0.0~20170104233230-1) over (2.0.0~20170103232707) ...
Processing triggers for systemd (215-17+deb8u5) ...
Setting up openhab2-offline (2.0.0~20170104233230-1) ...

And were done, upgraded and it seems to be working! a further sudo apt-get update told me I was already up to date too.

[theoweiss]

I also was also curious if it would work. I've tested an installation (raspberry pi 3) of "unstable" and an upgrade to "testing". Looks good!

[theoweiss]

High five @BClark09  

[BClark09]

🎉

[kaikreuzer]

Well done, guys!

So we will have to ask people to switch to the new Bintray repository, right? I am wondering, if we should combine this request with another change: While you were working on the packaging, I was busy last night restructuring the distro, so that we get rid off online/offline and will only have a single one. Additionally, there would then be an "addons" package that can be (optionally) downloaded to make everything available offline as well. This clearly will require some further changes to the debian package as well as communication to the users. I'll probably have my side ready this weekend. Wdyt?

[BClark09]

Yes, I can prepare a PR for the docs with the instructions above to apt-get from bintray.

To clarify, you're planning on keeping just the openhab2-online package (perhaps renaming it to "openhab2" and allowing users to download a seperate package containing all the addons if required? How will this addons package work? If all that is required is to have files in a particular folder then in debian package terms, we could add a separate openhab2-addons package which installs them for a user.

Otherwise is it possible to add an "offline=true" configuration for openHAB which downloads the addons on first launch? Changing the default false to true would be all that is necessary to change the openHAB2 install from an online distribution to an offline distribution.

[kaikreuzer]

I have just pushed my (WIP) current version to https://github.com/openhab/openhab-distro/pull/369.

Yes, for the manual download, there will be only one distro, simply openhab-2.0.0.zip, which is the online variant. Additionally, there is a addons-2.0.0.kar, which is a KAR (Karaf Archive) that contains all add-ons. This kar file can be put in the "addons" folder of the distro and is automatically picked up by it.

add an "offline=true" configuration for openHAB which downloads the addons on first launch

Haha, this somehow contradicts each other, doesn't it?

in debian package terms, we could add a separate openhab2-addons package which installs them for a user.

Yes, I think this would be the best. I'll probably also add a addons-legacy.kar, which contains all the legacy 1.x add-ons (which are currently only available online, but I think we can make them easily available through this mechanism as well).

[BClark09]

Haha, this somehow contradicts each other, doesn't it?

Eek, I didn't have my thinking cap on! :laughing:

That's fair enough, does openHAB need to be restarted when the KAR files are updated? Just thinking of how the new Debian package should be created. @theoweiss do you think there will be few issues in adding such a package?

The changes seen to the normal Linux user are pretty straight forward but I don't think there's a reasonable way of updating a package name, therefore I'm hoping that all the user has to do is:

sudo apt-get remove openhab2-o*
sudo apt-get install openhab2

And for people still wanting the "offline" approach, finish with:

sudo apt-get install openhab2-addons

This way, a sudo apt-get upgrade will update either when necessary. Speaking of which, how often do you expect to publish new kar files? I'm a little worried about how often we might publish to bintray.

[kaikreuzer]

does openHAB need to be restarted when the KAR files are updated?

No.

how often do you expect to publish new kar files?

Same as new distro, it comes out of the same build. I also publish the snapshot version to Bintray, but I always overwrite the previous version of it, so the size won't grow.