search

openhab / openhab-linuxpkg

Repo for Linux packages
Eclipse Public License 2.0
18 stars 33 forks source link

Initial work #2

Closed theoweiss closed 7 years ago

theoweiss commented 7 years ago

I've opened a PR for the initial checkin #1. I want to use this issue to discussing the further development. Here is a to-do list with the next things which have to be done. For sure the list is not complete, just add further to-dos if they are necessary.

Todo list for openhab b5

Todo list for openhab 2.0.0

theoweiss commented 7 years ago

Yes automatically removing old versions would be right solution, but for now I think it would be OK to remove them by hand using the web ui of bintray.

BClark09 commented 7 years ago

To achieve that I'd assume it would suffice to run a script to perform the following steps:

  1. Get a list of versions currently in the repo by parsing the response of: CURL -X GET "-u${BINTRAY_USER}:${BINTRAY_KEY}" "https://api.bintray.com/packages/${BINTRAY_ORG}/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/
  2. List the bottom [total number of versions] - [desired number of versions] that do not match "2.0.0~b*"
  3. For each match run: CURL -X DELETE "-u${BINTRAY_USER}:${BINTRAY_KEY}" "https://api.bintray.com/packages/${BINTRAY_ORG}/versions/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/${versionToDelete}
theoweiss commented 7 years ago

These curl calls from gradle are a bit ugly especially the error handling and even more if some complex logic is needed. Having these calls in an "external" bash script is a way to go, but I would prefer to have these things in the gradle build script. I want to give this rest-gradle-plugin a try and if I don't succeed we could take the bash script approach. What do you think?

ThomDietrich commented 7 years ago

Everything seems to work flawlessly till now. Great ;) Would it be possible to release a stable dummy package (e.g. the beta5 re-branded, needs to be "newer") so I can already test switching around between branches?

theoweiss commented 7 years ago

For testing the best way to go is:

ThomDietrich commented 7 years ago

Thanks for the instructions Theo but I do not intent to get active in the repository development. I've already too much on my plate, not only related to openHAB. I thought this would be a no-brainer for you guys and it wouldn't harm anyone. But it's okay, I'll just wait for the stable to be released.

BClark09 commented 7 years ago

I want to give this rest-gradle-plugin a try and if I don't succeed we could take the bash script approach. What do you think?

I would also like to have everything running from within the gradle build. I particularly like the responseHandler.

A quick example:

task "upload-${dist}"(type: org._10ne.gradle.rest.RestTask) {

  def fileName = "${gPackageName}_${gVersion}-${gRelease}_${OSPACKAGE_ARCH}.deb"

  httpMethod         = 'PUT'
  uri                = 'https://api.bintray.com/content/${BINTRAY_ORG}/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/${gVersion}/pool/main/${gVersion}/${fileName}'
  username           = ${BINTRAY_USER}
  password           = ${BINTRAY_KEY}
  requestBody        = new File("build/distributions/${fileName}")
  requestContentType = groovyx.net.http.ContentType.BINARY
  contentType        = groovyx.net.http.ContentType.JSON

  requestHeaders     = [X-GPG-PASSPHRASE:              ${BINTRAY_GPG},
                        X-Bintray-Debian-Distribution: ${gDebDist},
                        X-Bintray-Debian-Component:    'main',
                        X-Bintray-Debian-Architecture: ${BINTRAY_ARCH},
                        X-Bintray-Publish:             1]

  responseHandler = {
    assert it.message == 'success'
  }
}

But how to add the file to the body? I couldn't be sure.

theoweiss commented 7 years ago

You could try:

 requestBody = new File(fileName)

for the body.

And:

contentType = BINARY

I googled for "groovy RESTClient PUT file" because noamt uses "groovyx.net.http.RESTClient". Just a wild guess.

BClark09 commented 7 years ago

Yes, you're absolutely right! I've edited the above to contain a full example. Unfortunately, my trial expired so am not able to test what it does to Bintray.

theoweiss commented 7 years ago

Which trial? You don't have to pay for Bintray "Community - Open Source".

BClark09 commented 7 years ago

Whoops, my mistake. Created an account.

I've been struggling with this for a while, this is what I have so far (including a little tidy up): https://github.com/BClark09/openhab-linuxpkg/blob/add-rest-plugin/build.gradle

The request body needs to be converted to bytes to be used in requestBody (There's an issue for support of File), Since task configuration is done at the evaluation stage in gradle, I can't for the life of me work out how to use .bytes...

theoweiss commented 7 years ago

IMHO using .bytes may be a bad idea because I think the whole contents of the deb file would be stored in the requestBody variable, which means in RAM. Could you point me to the issue you mentioned?

BClark09 commented 7 years ago

Certainly - https://github.com/noamt/rest-gradle-plugin/issues/26

BClark09 commented 7 years ago

I've come to the conclusion that this won't accept a file stream for a long time. I had a play around with the underlining HTTP and REST clients and the resulting task was:

import groovyx.net.http.HTTPBuilder
import org.apache.http.entity.FileEntity
import org.apache.http.HttpHeaders

import static groovyx.net.http.ContentType.JSON
import static groovyx.net.http.Method.*
import static java.io.File.separatorChar
import static java.lang.System.err
import static java.lang.System.exit

buildscript {
    repositories {
        mavenCentral()
        jcenter()
    }   
    dependencies {
        classpath 'org.codehaus.groovy.modules.http-builder:http-builder:0.5.2'
    }   
}

task "upload-${dist}"(dependsOn: "build-${dist}") {

    def fileName = "${gPackageName}_${gVersion}-${gRelease}_${OSPACKAGE_ARCH}.deb"
    def fileToUpload = new File("build/distributions/${fileName}")

    HTTPBuilder bintray = new HTTPBuilder('https://api.bintray.com', JSON)
    bintray.parser.'application/json' = { '' }
    bintray.auth.basic "$BINTRAY_USER", "$BINTRAY_KEY"

    def requestHeaders = [
        'X-GPG-PASSPHRASE':              "${BINTRAY_GPG}",
        'X-Bintray-Debian-Distribution': "${gDebDist}",
        'X-Bintray-Debian-Component':    'main',
        'X-Bintray-Debian-Architecture': "${BINTRAY_ARCH}",
        'X-Bintray-Publish':             '1'
     ]

    doLast{
        bintray.request(PUT, JSON) {
            uri.path = "/content/${BINTRAY_ORG}/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/${gVersion}/pool/main/${gVersion}/${fileName}"
            headers[HttpHeaders.AUTHORIZATION] = 'Basic ' + ("$BINTRAY_USER:$BINTRAY_KEY".toString().bytes.encodeBase64())
            requestHeaders.each { key, value ->
                headers."${key}" = "${value}"
            }
            contentType = groovyx.net.http.ContentType.JSON
            requestContentType = groovyx.net.http.ContentType.BINARY
            body = fileToUpload.bytes
            response.success = {
                println 'Done.'
            }
            response.'409' = {
                err.println "WARNING: File already uploaded, new file ignored."
            }
            response.failure = {
                err.println "Failed to upload file: $it.statusLine.reasonPhrase."
                exit it.status
            }
        }    
    }
}

I think i'd prefer to use the curl executable task within gradle considering the memory usage in this task and the extra faff that needs to be performed to get it to function correctly. I'd also prefer to use the existing task over a bash script, wdyt?

theoweiss commented 7 years ago

Your right. I think the bug is in groovyx.net.http library and we would have to use the apache hc lib to get a proper solution. For now we should stick with the current curl way. What we still need is a task for removing outdated versions, where we don't have the body problem. We could use groovyx.net.http for this or a well crafted bash script.

But on the top of the todo list is the adaption of the gradle script to release 2.0.0.RC1 see here: https://community.openhab.org/t/upcoming-releases/18503/17

theoweiss commented 7 years ago

@BClark09 I've started to work on the adaptions needed for releasing RC1.

BClark09 commented 7 years ago

Right, is RC1 going in Stable or Testing? Apologies, I've not had any notification on that message chain until today so was not privy to the release dates.

I was in the process of cleaning up the build.gradle (see branch here), but I can merge this after RC1 and also work on a way of cleaning up the old snapshot versions. I think we should be fine using the REST plugin here.

@theoweiss Would you be able to add 'userdata/etc/branding-ssh.properties' to the list of files that we dis-include from the conffile list? This came up as a problem for some people using apt.

kaikreuzer commented 7 years ago

Right, is RC1 going in Stable or Testing?

I'd say Testing, just like the beta releases.

theoweiss commented 7 years ago

@BClark09 I'm just making the RC1 release. I've merged the necessary changes to this project. I'll do a test installation. The distribution name is "testing". If you have time tomorrow could you also make some basic tests?

theoweiss commented 7 years ago

@theoweiss Would you be able to add 'userdata/etc/branding-ssh.properties' to the list of files that we dis-include from the conffile list? This came up as a problem for some people using apt.

@BClark09 now I've not been notified about your comment. We could add it to the next release or even increment deb "release" number and make another deb release for RC1.

I'm sure you were not aware of this community thread (https://community.openhab.org/t/upcoming-releases/18503/17m) because you haven't been in the maintainer list at the time this thread was opened.

With a next step we have to add a parameterised task for releasing 2.x.x., shouldn't be to hard to implement.

BClark09 commented 7 years ago

That's fine, upgrade on a debian 8 machine is working fine. Also tested a fresh install on Ubuntu 14. The packages are doing their job fine and addons seems to be working fine also.

However, @kaikreuzer, after the install of legacy, 1.x bindings show up on paper-ui, but when trying to install them they stall and on refresh they no longer show, even on a restart, this behaviour is seen even on a fresh install, can you and @theoweiss can confirm this behaviour?

I'm sure you were not aware of this community thread (https://community.openhab.org/t/upcoming-releases/18503/17m) because you haven't been in the maintainer list at the time this thread was opened.

Initially it read "Oops! That page doesn’t exist or is private." but I can read this on refresh. Thanks.

kaikreuzer commented 7 years ago

can you and @theoweiss can confirm this behaviour?

I tried the "normal" install and all seems to work well:

08:02:06.310 [INFO ] [smarthome.event.ExtensionEvent      ] - Extension 'binding-astro1' has been installed.
08:02:20.295 [INFO ] [b.core.service.AbstractActiveService] - Exec Refresh Service has been started
08:02:20.298 [INFO ] [smarthome.event.ExtensionEvent      ] - Extension 'binding-exec1' has been installed.

So I assume it is either some local problem on your end or some issue with the apt package.

BClark09 commented 7 years ago

@kaikreuzer You're right, it was a local problem and works as expected now. But I am still getting some minor behaviour:

When the legacy switch is off in paper UI and then I add the legacy .kar file, (through apt, or normal installation) all the legacy bindings show up only once in the add-ons menu. If you try to install these, they will disappear until you actually turn on the legacy switch, and installation of these bindings works as expected.

kaikreuzer commented 7 years ago

Ok, I can live with that ;-)