search

openhab / openhab-linuxpkg

Repo for Linux packages
Eclipse Public License 2.0
18 stars 33 forks source link

Initial work #2

Closed theoweiss closed 7 years ago

theoweiss commented 7 years ago

I've opened a PR for the initial checkin #1. I want to use this issue to discussing the further development. Here is a to-do list with the next things which have to be done. For sure the list is not complete, just add further to-dos if they are necessary.

Todo list for openhab b5

Todo list for openhab 2.0.0

BClark09 commented 7 years ago

Fantastic, you've been busy! Are the changes in the PR ready for an initial test? Am I right in thinking ./gradlew download-openhab2-online-snapshot && ./gradlew distro-openhab2-online-snapshot is the correct order to get a .deb file?

update docs

From a user's point of view, what will be different to how installation usually works which requires the documentation to change?

theoweiss commented 7 years ago

I've just updated the README: ./gradlew buildSnapshot is your friend ;-).

From a user's point of view, what will be different to how installation usually works which requires the documentation to change?

The snapshot installation will probably change because I'll try to publish the snapshots using the bintray apt-repo.

theoweiss commented 7 years ago

@BClark09 I've merged the initial checkin. It would be great if you could do some testing.

BClark09 commented 7 years ago

Sure thing! I've got a spare Debian laptop ready for testing, but will get round to doing it as soon as I can.

ThomDietrich commented 7 years ago

Hey @theoweiss these points are of concern to me:

  • release snapshots to bintray (replacement for apt-repo plugin)
  • integrate snapshot build and release into cloudbees
  • update docs

Will identical snapshots be available on both bintray and cloudbee? If so, how long? If I understand you correctly, the idea is to provide the latest snapshot on both to ease the transition for users?

What I take from that: I'll have to switch the repository link to bintray in both the linux article and the openHABian scripts!?

BClark09 commented 7 years ago

Testing went well:

  1. I used ./gradlew buildSnapshot to build both the OFFLINE and ONLINE .deb packages of snapshot #624. No apparent error in building.

  2. I installed openHAB on a fresh debian laptop using the offline .deb package first, no errors were logged in dpkg, or the service daemon on first run. openHAB configured itself successfully without error.

  3. I installed the NTP, Yahoo Weather, and Exec1.x bindings, added some sample things and items and used mapDB persistence to restoreOnStartup. Worked flawlessly.

  4. Forced an upgrade using the same deb package, after "upgrade" bindings reinstalled themselves and persisted items displayed the expected state.

  5. Repeated Steps 1-4 with the online package.

  6. Repeated Steps 1-5 with the cloudbees packages, making sure to upgrade to the packages built earlier. Once again, nothing unexpected popped up during testing. Only comment here is that builds from gradle have -1 appended to the timestamp.

I'm going to build #625 tomorrow so that I can test the above with specifically different snapshot files but I don't expect any differences in behaviour so from what I can see so far:

Fresh installation works. Cloudbees snapshot > gradle snapshot upgrade works. gradle snapshot > gradle snapshot upgrade works.

theoweiss commented 7 years ago

@BClark09 you(!) have been busy, thanks a lot. Sounds very good. We also have to check the issues for the deb packaging opened in the openhab-distro project. I found : https://github.com/openhab/openhab-distro/issues/328

theoweiss commented 7 years ago

@ThomDietrich

Will identical snapshots be available on both bintray and cloudbee? If so, how long? If I understand you correctly, the idea is to provide the latest snapshot on both to ease the transition for users? What I take from that: I'll have to switch the repository link to bintray in both the linux article and the openHABian scripts!?

I hope to get the bintray deployment ready before the b5 release. If I get this working the snapshots will be available for a short time on both, bintray and cloudbees to ease the transition. Together with the b5 release we will remove the snapshot build from the openhab-distro project. Hopefully. I've read the available docs how to release from gradle to bintray but found no docs or examples how to handle debian repos. As you can see there is some uncertainty if all these things are in place until the b5 release.

BClark09 commented 7 years ago

@theoweiss

First major problem I'm coming up with. It doesn't look like there's any handling of configuration files even though I see they're set in gradle. For example my changes to /etc/openhab2/html/index.html are lost on an upgrade when using dpkg.

BClark09 commented 7 years ago

It looks like this issue is related to: https://github.com/nebula-plugins/gradle-ospackage-plugin/issues/56

EDIT 1: Adding the line "configurationFile("/etc/openhab2/html/index.html")" in ospackage {.. stops replacing the html file, but obviously this is going to be a problem for us with the number of configuration files we have. Wildcards did not work.

EDIT 2: Adding "configurationFile(details.path)" in eachFile might work, will test and submit a PR if true.

EDIT 3: Seems good, submitted PR #3

BClark09 commented 7 years ago

Alright! So Debian looks good to go,

We also have to check the issues for the deb packaging opened in the openhab-distro project. I found : openhab/openhab-distro#328

The build doesn't create a /var/run/openhab.pid so I'm guessing this is Ubuntu specific, I'll install Ubuntu on a VM and get back to you on this. https://github.com/openhab/openhab-distro/issues/327 is also relevant.

ThomDietrich commented 7 years ago

I believe the pid file problem is related to the systemd service file openhab2.service. http://www.dsm.fordham.edu/cgi-bin/man-cgi.pl?topic=systemd.service -> PIDFile and GuessMainPID These options are not in the files, which may be part of the problem and part of the solution...

BClark09 commented 7 years ago

Oh right, fair enough thanks. Then it'll be a mixture of that assigning start.sh as it's main program and init.d/openhab2 using start-stop-daemon and --exec

BClark09 commented 7 years ago

Whilst looking into the above, I notice that /etc/init.d/openhab2 isn't included in the gradle build. I've got a branch I'm working on to address these two issues.

BClark09 commented 7 years ago

@theoweiss I've tested this on multiple Linux distributions now (init.d systems need PR #6 included) across multiple updated and I'm pretty confident in the Initial work. What are the next steps here? A release task for bintray?

theoweiss commented 7 years ago

@BClark09 I've just merged your PR, thanks again.

Yes we will need two release tasks for bintray: one for the snapshots and one for the releases. As soon as the linux-pgk build is integrated into the cloudbees build the snapshot build must be triggered automatically each time when the openhab-distro build finished successfully.

I think we should try to use the bintray gradle plugin (https://github.com/bintray/gradle-bintray-plugin). Unfortunately I found no examples for publishing deb files. Here you will find the script that I've used for publishing openHAB2 beta releases: https://github.com/openhab/openhab-linuxpkg/blob/master/resources/bin/bintray-upload-debs.sh . This must be transcribed to gradle-bintray and for snapshot releases.

Furthermore I've started to port the apt-repo maven plugin to gradle (https://github.com/theoweiss/apt-repo). Until now we used it for publishing the snapshot debs. You will find my work here https://github.com/theoweiss/apt-repo-lib/tree/initialDev and here https://github.com/theoweiss/apt-repo-gradle. The idea is to have this available in case we can't get the gradle-bintray plugin running.

If you don't mind it would be great if you could care for the gradle-bintray task and I'll try to finish the apt-repo gradle plugin as a fallback solution?

BClark09 commented 7 years ago

Not sure when I'll be able to look at it, but I did previously have a look and think that this will get us pretty close:

bintray {
  user = System.env.BINTRAY_USER
  key = System.env.BINTRAY_API_KEY

  dryRun = false
  publish = true    
  override = true

  filesSpec {
    from ('build/distributions') { include '*.deb' }
       into '/'
  }

  pkg {
    repo = 'apt-repo2'
    name = 'openhab2'
    userOrg = 'openhab'
    desc = 'Packages for openHAB 2.x'
    websiteUrl = 'http://www.openhab.org/'
    issueTrackerUrl = 'https://github.com/openhab/openhab-linuxpkg/issues'
    licenses = ['EPL-1.0']
    vcsUrl = 'https://github.com/openhab/openhab-linuxpkg'

    attributes = ['deb_distribution': 'development', 'deb_component': 'main', 'deb_architecture': 'all']
   }
}

Although I'm a little unsure how it will actually handle the different deb files. It'll probably need a version parameter with a variable name parameter that can be set to [online|offline]-[snapshot|beta] but not sure how best to do that yet.

BClark09 commented 7 years ago

Hi @theoweiss, I've edited the above, but am very unsure on if this would generate anything useful. Would you mind testing this? (Probably best into the test name instead of openhab2). 

.gradlew bintrayUpload

Initial thoughts:

EDIT: Nope, got a trial account and can't get this working at all. Any iteration of it and I'm stuck with the error:

message: Unable to upload files: Debian distribution. component and architecture must be specified.

As far as I know, the attributes should be set according to https://bintray.com/docs/api/

theoweiss commented 7 years ago

I had a quick look into the sources of gradle-bintray-plugin. As far as I understand the attributes will be used to create the request body: https://github.com/bintray/gradle-bintray-plugin/blob/58a2c1813ebadc8ad3242a6ca73a4fa1471823c5/src/main/groovy/com/jfrog/bintray/gradle/BintrayUploadTask.groovy#L231 . But the documentation indicates that debian information must be specified using HTTP matrix parameters or HTTP request headers (https://bintray.com/docs/api/#_debian_upload). May be using the gradle-bintray-plugin is a dead end approach :-(

theoweiss commented 7 years ago

May be there are two other options:

  1. gradle Exec tasks https://docs.gradle.org/current/dsl/org.gradle.api.tasks.Exec.html and make a curl call similar to that one in the upload script https://github.com/openhab/openhab-linuxpkg/blob/master/resources/bin/bintray-upload-debs.sh .

  2. the gradle rest plugin https://github.com/noamt/rest-gradle-plugin

We would also need a task to add a new version for each upload.

BClark09 commented 7 years ago

I'd also seen the gradle Exec task as this seems to be the one used in most deb builds (for example). I like the idea of using the REST plugin though. I might have the chance to play with both tomorrow!

BClark09 commented 7 years ago

@theoweiss , What do you think of https://bintray.com/benclark as a way of managing these packages. Check out the apt-snapshot repo where I've managed to get version uploads working, we may also need to add another task to clear old versions. I can successfully install and upgrade using apt when I add the repo to the sources list too, it now recognises when you're already on a version!

My own branch now does this through gradle as part of the buildSnapshot command, or manually start an upload with upload-openhab2-online-snapshot etc.

theoweiss commented 7 years ago

@BClark09 I'm preparing some feedback until tomorrow.

BClark09 commented 7 years ago

@theoweiss, did you have any quick points of feedback? I have time to work on it this weekend.

theoweiss commented 7 years ago

Hi @BClark09,

sorry for the delay! First of all I think I should explain the apt-repo setup I had in my mind. I think we should have only one repo, which is the still existing apt-repo2 for openHAB2 (https://bintray.com/openhab/apt-repo2/openhab2). All released packages should be available from this repo. Released packages are snapshots, betas and stable versions. Having more than one repo until now is just a workaround for technical reasons.

user = project.hasProperty('bintrayUser') ? project.property('bintrayUser') : System.getenv('BINTRAY_USER')

Regards, Theo

BClark09 commented 7 years ago

Thanks @theoweiss I've started updating it now, does https://bintray.com/benclark/apt-repo2 look a little more like what you were expecting (distribution needs to be changed from "null" to "testing")? Or would you rather everything be placed inside an openHAB2 package?

theoweiss commented 7 years ago

dists should not be null: https://bintray.com/benclark/apt-repo2/openhab2-offline#files/dists Yes place it into the openHAB2 package and separate it by versions.

BClark09 commented 7 years ago

Alright thanks once again @theoweiss, I think we're on the same page now, sorry for the trial-error approach:

https://bintray.com/benclark/apt-repo2/openhab2

theoweiss commented 7 years ago

Cool, this looks very promising.

Now we have to tweak the deb creation a little bit:

sorry for the trial-error approach

It's a good agile approach and helped to continue the discussion.

kaikreuzer commented 7 years ago

maintainer should be "openHAB admin@openhab.org"

No, this actually goes nowhere really... Does a Debian package NEED to provide an e-mail address of a maintainer? If not, I would suggest to put a link like https://community.openhab.org/ as a reference for support.

ThomDietrich commented 7 years ago

Is it actually still beneficial to provide online and offline versions of openHAB? The differences are clear to us (complete but more used space vs. internet connection needed) but maybe not clear and probably not relevant to the end user.

From the end users perspective two non-default packages are in my opinion confusing. How about either providing a default and the other (e.g. openhab2+openhab2-offline or similar) or going the path many other repositories/package maintainers go: Defining a meta package "openhab2" pointing to the actual package openhab2-online.

Wdyt?

BClark09 commented 7 years ago

Alright! I've got a working version with changes @theoweiss asked of me: https://bintray.com/benclark/apt-repo2/openhab2 with the maintainer set as suggested by @kaikreuzer.

The description seems to be altering itself by adding on the additional properties of the Debian build, I think this might be a bug in ospackage but I think we should leave as is for now.

This is working very well when I include the following into a sources file.

deb https://benclark.bintray.com/apt-repo2 unstable main

apt-get will recognise same versions, and will not attempt to reinstall unless forced. It will also happily upgrade from a cloudpath version.

@ThomDietrich I like the idea, but which package do we even label as default? I have a feeling that around the world they would be used equally.

BClark09 commented 7 years ago

What is bintray's support for snapshots? It looks they don't have an official quota for snapshots as such but the more I look into keeping the repo clean, the more I see suggestions for using bintray for releases only.

kaikreuzer commented 7 years ago

Bintray does not allow snapshots. They instead go to https://oss.jfrog.org.

BClark09 commented 7 years ago

Right, so the work in #9 will need to be changed so that gradle uploads the snapshots to the oss artifactory, and the beta releases to bintray?

theoweiss commented 7 years ago

9 looks good so far. I'll merge the PR because releasing betas should work. The snapshot issue could be solved with a further PR.

theoweiss commented 7 years ago

@kaikreuzer could you provide us some details about openHAB on oss.jfrog.org.

kaikreuzer commented 7 years ago

You can find infos here: https://www.jfrog.com/confluence/display/RTF/Deploying+Snapshots+to+oss.jfrog.org

tl;dr: When Bintray packages are also published to JCenter (what we do), the Bintray credentials will automatically also work for oss.jfrog.org, so that you can use that repo in your Maven pom as a deploy repo.

kaikreuzer commented 7 years ago

See e.g. https://github.com/openhab/openhab-distro/blob/master/pom.xml#L40

ThomDietrich commented 7 years ago

@kaikreuzer Any opinion on what I said earlier? For openHAB 2.0 GA I'd reevaluate if the online or offline version should be installed by openHABian by default and I am not even sure which one to recommend at the moment.

kaikreuzer commented 7 years ago

@ThomDietrich I agree that for openHABian we should define which version to use. I personally prefer the offline distro very much - installation also works offline and all the setup is much quicker and less error-prone. If size is not a matter (and 16GB SD cards is the smallest you can buy nowadays...), I would go for this. As a side note: For the Ubuntu Core Snap, we decided to use the online distro as Snaps are anyhow remotely updated and installed.

Thinking further into the future, we might also one day merge the two versions into one: Have the most common add-ons always shipped with the distro, while having the million others installable from a remote repo.

ThomDietrich commented 7 years ago

@kaikreuzer From your answer I get the idea that your answer to my other question would be "leave as is for now"!?

kaikreuzer commented 7 years ago

I didn't really understand what you meant with "default" packages there, so no, I tried to avoid answering that question as I didn't understand it ;-)

ThomDietrich commented 7 years ago

^^ touché. I was trying to propose to promote one version as the favorable one. Currently a user can select either the online or offline version but the pros and cons are unclear to users and I feel they shouldn't have to care. Hence the proposal to provide a deb package with the name "openhab2".

kaikreuzer commented 7 years ago

Well, then my answer still fits - I think there is only a recommendation per packaging/installation option. As said above, for snaps I would suggest "online" (that's actually the only one we offer), while for openHABian I would go for "offline".

If you want one recommendation for debian in general, I think it should be offline as well - this is what users usually expect from a debian package: That it is self-contained and does not require downloading a lot of additional stuff in order be usable, right?

ThomDietrich commented 7 years ago

I'm not sure about the last part (compare Firefox or other add-ons based software) but I get your argument "recommendation per packaging/installation option". I thought with the online version mostly stable this would be a good time to think about abandoning the offline version for openHAB 2.0 GA all together (or provide it as a less favorable version) but I do not want to argue about this in length here. Just wanted to point it out as this Issue / repo would be influenced by such a decision. ;)

theoweiss commented 7 years ago

@kaikreuzer it seems like OJO ( https://oss.jfrog.org) isn't a valid option because https://www.jfrog.com/confluence/display/RTF/Deploying+Snapshots+to+oss.jfrog.org states "hosting your maven-compatible build snapshots". Debian repos which are not maven-compatible don't seem to be supported :-( Some time ago you suggested to use bintray for the nightly builds because the version string doesn't contain "SNAPSHOT" anymore: https://github.com/openhab/openhab-distro/issues/270#issuecomment-244405155 From a technical perspective this would work.

theoweiss commented 7 years ago

As a fallback we could also stick with cloudbees and use the gradle port of the maven apt-repo plugin. It would be still better than the current solution because we could solve the lastSuccessfulBuild/WS problem which is also mentioned here https://github.com/openhab/openhab-distro/issues/270#issuecomment-244405155

kaikreuzer commented 7 years ago

it seems like OJO ( https://oss.jfrog.org) isn't a valid option

Hm, good point. I didn't think about the fact that we are not talking about Maven artifacts.

Some time ago you suggested to use bintray for the nightly builds because the version string doesn't contain "SNAPSHOT" anymore

Let's go for this. After all, Debian packages do not know the notion of a "snapshot", so it should be fine to upload all of them to Bintray.

BClark09 commented 7 years ago

Alright! That should be fine but I think we will need a task to remove old versions.