search

openhab / openhab-distro

The binary distribution of openHAB
https://www.openhab.org/
Eclipse Public License 2.0
1.3k stars 394 forks source link

Openhab2 ubuntu snapshot PPA not working. #270

Closed skaggapa closed 7 years ago

skaggapa commented 7 years ago

Hi!

Added ppa according to instructions on docs.openhab.

Get the following error...

E: Failed to fetch https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo/Release  Unable to find expected entry 'Packages' in Release file (Wrong sources.list entry or malformed file)
E: Failed to fetch https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo/Release  Unable to find expected entry 'Packages' in Release file (Wrong sources.list entry or malformed file)
E: Some index files failed to download. They have been ignored, or old ones used instead.

My sources.list.

deb https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo/ /
deb https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo/ /
kaikreuzer commented 7 years ago

@theoweiss Could you comment?

ThomDietrich commented 7 years ago

This is a problem occurring on Ubuntu 16.04 and probably other systems with recent apt packages.

Also discussed here https://community.openhab.org/t/oh2-via-apt-get-no-habamin-ui/13305/20

theoweiss commented 7 years ago

Just a first guess: This could happen if the last build failed or if a build is currently running. The repo files are in a subdirectory of the "target" directory (the maven ${buildDirectory}) which is removed at the beginning of a build and will be recreated at the end of a successful build. This could be improved by configuring a repoDir (docs are here https://github.com/theoweiss/apt-repo) which won't be removed by the build. @kaikreuzer is an appropriate directory available on cloudbees?

If this was the cause the error should recover, especially it should work at the moment.

theoweiss commented 7 years ago

@ThomDietrich signing the repo is currently not implemented by the apt-repo maven plugin. I think it should not be to hard to implement this, but it only makes sense if the is a secure and protected place for the gpg key on cloudbees. @kaikreuzer does cloudbees provide a place like that?

ThomDietrich commented 7 years ago

Hey @theoweiss, thanks for the insight. I'm not sure about your first guess as the error is permanently available on a Ubuntu 16.04 installation while I'm able to update on a Raspberry Pi (jessie). The complete error message: https://community.openhab.org/t/oh2-via-apt-get-no-habamin-ui/13305/7

theoweiss commented 7 years ago

May be 16.04 wants SHA512 hashes in the Release file. I'll check this.

skaggapa commented 7 years ago

I have had this problem for two days. I didnt report it until today because i know how snapshot repos are prone to temporary errors.

Some more info from about my system.

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

Linux hades 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

kaikreuzer commented 7 years ago

The repo files are in a subdirectory of the "target" directory (the maven ${buildDirectory}) which is removed at the beginning of a build and will be recreated at the end of a successful build.

Err, the "ws" should indeed never be referenced, because as you say it can be empty or even contain files from a failed build. So in general, the build plan must be configured to archive artifacts, which are then ALWAYS available at https://openhab.ci.cloudbees.com/job/openHAB-Distribution/lastSuccessfulBuild I am not sure whether it is possible to archive folders though, usually these are only files, but this is probably anyhow not the right approach, see the next question.

protected place for the gpg key on cloudbees. @kaikreuzer does cloudbees provide a place like that?

Signing of the release packages is done on bintray and @theoweiss has set all of this up there. @theoweiss As with https://github.com/openhab/openhab-distro/pull/267 we do not have SNAPSHOT-debs anymore, would it be possible to simply also automatically deploy the snapshot builds to bintray? Maybe it is helpful to have a look at https://github.com/openhab/openhab-deps-repo/blob/master/p2/org.openhab.deps.repository/src/main/filtered-resources/publish-bintray.sh - this is also executed on cloudbees and automatically uploads an unzipped p2 repo on bintray; this works pretty smooth!

theoweiss commented 7 years ago

Maybe it is helpful to have a look at https://github.com/openhab/openhab-deps-repo/blob/master/p2/org.openhab.deps.repository/src/main/filtered-resources/publish-bintray.sh - this is also executed on cloudbees and automatically uploads an unzipped p2 repo on bintray; this works pretty smooth!

@kaikreuzer how is the execution of this script triggered?

kaikreuzer commented 7 years ago

Simply through a script execution task in the Jenkins build plan configuration.

theoweiss commented 7 years ago

Do I have access to this configuration?

kaikreuzer commented 7 years ago

At least now you do - I added you as an admin to cloudbees :-)

theoweiss commented 7 years ago

Thanks @kaikreuzer .

I can confirm 16.04 does not work even if --allow-unauthenticated is used. I've enhanced the apt-repo plugin to generate SHA512 hashes but it still doesn't work. Next I will try to add gpg signing to the apt-repo plugin. And in medium-term I will try to host the snapshots at bintray.

ThomDietrich commented 7 years ago

as an in between statement: Thank you @theoweiss for your efforts!

skaggapa commented 7 years ago

Hi!

The error is different now.

Ign:9 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  InRelease
Ign:10 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  InRelease
Ign:11 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Release
Get:12 http://se.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [378 kB]
Get:13 http://se.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [323 kB]
Get:14 http://se.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [320 kB]
Ign:15 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Release
Ign:16 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Packages.diff/Index
Ign:17 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en_US
Ign:18 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en
Ign:19 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Packages.diff/Index
Ign:20 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en_US
Ign:21 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en
Ign:22 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Packages
Ign:17 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en_US
Ign:18 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en
Ign:23 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Packages
Ign:20 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en_US
Ign:21 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en
Ign:22 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Packages
Ign:17 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en_US
Ign:18 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en
Ign:23 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Packages
Ign:20 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en_US
Ign:21 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en
Ign:22 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Packages
Ign:17 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en_US
Ign:18 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en
Ign:23 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Packages
Ign:20 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en_US
Ign:21 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en
Get:22 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Packages [337 B]
Ign:17 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en_US
Ign:18 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en
Get:23 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Packages [461 B]
Ign:20 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en_US
Ign:21 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en
Ign:17 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en_US
Ign:18 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Translation-en
Ign:20 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en_US
Ign:21 https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Translation-en
Fetched 1 500 kB in 7s (211 kB/s)                                              
Reading package lists... Done
W: The repository 'https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-offline/target/apt-repo  Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'https://openhab.ci.cloudbees.com/job/openHAB-Distribution/ws/distributions/openhab-online/target/apt-repo  Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
theoweiss commented 7 years ago

Until now the pom and cloudbees configuration is unchanged. I've worked on the apt-repo plugin and added signing capabilities. While working on this my experience with apt was that the error messages are quite misleading and the behavior of apt is a little bit unpredictable. The next steps are publishing the new apt-repo plugin to maven central and then tweaking the maven and jenkins configuration.

otbe commented 7 years ago

Hi I just get 

Reinstallation of openhab2-online is not possible, it cannot be downloaded.

from sudo apt-get --reinstall install openhab2-online. Is this related to this issue? sudo apt-get update works without problems :)

ThomDietrich commented 7 years ago

Hello @otbe, this is not related. a normal apt update && apt upgrade should be enough. "--reinstall" is not needed anymore, see the note under http://docs.openhab.org/installation/linux.html#upgrade

otbe commented 7 years ago

It works as it should. Thanks!

theoweiss commented 7 years ago

FYI: The apt-repo 0.3.0 plugin is published and I've worked out a configuration with my local jenkins CI. All looks promising, even making the apt-repo directory available on lastSuccessfulBuild should be feasible.

pflodo commented 7 years ago

I think this is the same issue I am seeing on Debian (Stretch), and haven't been able to solve: https://community.openhab.org/t/repository-issue-on-debian-for-snapshot/13549

Note that untrusted sources can be added to your sources file with: deb [trusted=yes] URL

This got rid of the untrusted repository message, but it didn't solve my issue.

Making a proper signed repository however is an excellent way forward.

kaikreuzer commented 7 years ago

Can this be closed then?

theoweiss commented 7 years ago

Not yet. I haven't changed the Jenkins configuration until now, because I need some further testing of the lastSuccessfulBuild configuration.

theoweiss commented 7 years ago

The repo signing is in place! From now on users have to add the openHAB Snapshot signing key to their apt keyring if they want to use the snapshot repo.

curl 'http://www.openhab.org/keys/public-key-snapshots.asc' | sudo apt-key add -

@ThomDietrich it would be great if you would update the docs accordingly?

Unfortunately until now I had no success to add the apt repos in the right way to lastSuccessfulBuild. Adding worked but it would change the whole structure of the published files as you can see here: lastsuccessfulartifacts This is a no go because depended projects like openhab-docker would get into trouble.

kaikreuzer commented 7 years ago

@theoweiss Would you only need the deb file? Btw, why are there two deb files created, one in target and one in target/apt-repo?

theoweiss commented 7 years ago

@kaikreuzer with the current maven build I would need two directories one for openhab-offline/apt-repo and one for openhab-online/apt-repo with the contents like shown in the screenshot. The better solution would be to change the maven build in a way that only one apt-repo with both distros would be created. I've tried this but had no success due to my limited maven know how. Here you can see what I've tried: https://github.com/theoweiss/openhab-distro/commit/5cab295463427ee2b620a22d6af91a9587afe2ba

Btw, why are there two deb files created,..

The maven build creates only one deb file per distro but the Jenkins archiving rule which I've used created two copies distributions/openhab-*/target/*.zip,distributions/openhab-*/target/*.deb,distributions/openhab-*/target/*.tar.gz,distributions/openhab-*/target/apt-repo/*

kaikreuzer commented 7 years ago

The better solution would be to change the maven build in a way that only one apt-repo with both distros would be created.

Yes, this would indeed be much better! As I have limited apt-get-Maven-plugin know-how, I also cannot easily guide you on how to best get there...

ThomDietrich commented 7 years ago

Added to both docs and openHABian. Thanks @theoweiss !

One question: I tried the process on my openHABian testbed. It seemed to me, that the last upgrade switched over to b4 and my system was now not able to upgrade beyond that. I had to purge, clean and install to get the newest snapshot. This would be a quite important problem to inform users of / include a solution for in openHABian.

The situation after adding the key:

[13:24:11] pi@openHABianPi:~$ sudo apt search openhab2
Sorting... Done
Full Text Search... Done
openhab2-offline/now 2.0.0~b4 all [installed,local]
  openHAB Offline Distribution

openhab2-online/unknown 2.0.0~20160923010346 all
  openHAB Online Distribution

[13:24:34] pi@openHABianPi:~$ sudo apt install --reinstall openhab2-offline
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reinstallation of openhab2-offline is not possible, it cannot be downloaded.
0 upgraded, 0 newly installed, 0 to remove and 33 not upgraded.
ThomDietrich commented 7 years ago

Hopefully this will catch a few users: https://community.openhab.org/t/apt-snapshot-installation-notice/14407

theoweiss commented 7 years ago

Very much thanks for that! I think the community thread already clarified things. Reinstall won't work anymore since this PR was merged, because the snapshot versioning changed to timestamps: https://github.com/openhab/openhab-distro/pull/267

ThomDietrich commented 7 years ago

It did indeed. This is actually a good example, why we should nurture articles, tutorials and snippets in the community forum before posting them somewhere in the docs.

I do not think timestamps are the cause here but anyway, remove and install seems to solve all problems the guys in the forum thread had.

skaggapa commented 7 years ago

Just tested online distro and it works!

Thanks a lot!

theoweiss commented 7 years ago

I will close this issue because the main issue is fixed. For further optimizations regarding the lastSuccessfulBuild a new issue would be fine.