Allow transfer of configuration over from OH2 to OH3

[BClark09]

Closes #180 (DEFAULT not added as these files are additionally handled by DPKG/RPM, for apt users these files will ask the user what the user wants to do with the changed file, our recommendation should be "yes, change to maintainers version").

For the update to work as update.lst intends, the user must update to the latest stable version of openHAB 2.x first.

Tested on .deb and .rpm... repository hosting testing pending.

Signed-off-by: Ben Clark ben@benjyc.uk

[BClark09]

@openhab/distro-maintainers,

This PR allows the OH3 package to copy over OH2 configuration into the new file structure (**/openhab/** instead of **/openhab2/**). I've tested this by having an openHAB 2.5.8 install with two Astro Things which linked items all done in paperUI. After the install of openHAB 3, (which removes the openhab2 package) these items are still available and are updated within the new UI.

Unfortunately there are two issues, both to do with felix.fileinstall (I think).

1. paper and dashboard still exist in addons.config, this is only a problem because the runtime cycles after a Failed installing 'openhab-ui-dashboard, openhab-ui-paper' error and this cycle keeps stopping and starting all bundles in an infinite loop.

   /var/lib/openhab/config/org/openhab/addons.config:6:ui="dashboard,basic,paper,habpanel,homebuilder"

2. File install appears to have hardcoded the paths for the addons directory, which means that the runtime is trying to find files in /var/lib/openhab2 and /usr/share/openhab2 and will ignore the designated path for $OPENHAB_ADDONS:

   ben@debian:~/Git/openhab-linuxpkg$ grep -rnw '/var/lib/openhab' -e 'openhab2'
   /var/lib/openhab/config/jmx/acl/org/apache/karaf/config.config:18:felix.fileinstall.filename="file:/var/lib/openhab2/etc/jmx.acl.org.apache.karaf.config.cfg"
   /var/lib/openhab/config/jmx/acl/org/apache/karaf/bundle.config:2:felix.fileinstall.filename="file:/var/lib/openhab2/etc/jmx.acl.org.apache.karaf.bundle.cfg"
   /var/lib/openhab/config/org/apache/karaf/log.config:2:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.log.cfg"
   /var/lib/openhab/config/org/apache/karaf/features/repos.config:20:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.features.repos.cfg"
   /var/lib/openhab/config/org/apache/karaf/jaas.config:8:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.jaas.cfg"
   /var/lib/openhab/config/org/apache/karaf/command/acl/system.config:2:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.command.acl.system.cfg"
   /var/lib/openhab/config/org/apache/karaf/command/acl/scope_bundle.config:5:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.command.acl.scope_bundle.cfg"
   /var/lib/openhab/config/org/apache/karaf/command/acl/shell.config:3:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.command.acl.shell.cfg"
   /var/lib/openhab/config/org/apache/karaf/command/acl/feature.config:2:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.command.acl.feature.cfg"
   /var/lib/openhab/config/org/apache/karaf/command/acl/config.config:8:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.command.acl.config.cfg"
   /var/lib/openhab/config/org/apache/karaf/command/acl/kar.config:2:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.command.acl.kar.cfg"
   /var/lib/openhab/config/org/apache/karaf/command/acl/jaas.config:2:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.command.acl.jaas.cfg"
   /var/lib/openhab/config/org/apache/karaf/command/acl/bundle.config:2:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.karaf.command.acl.bundle.cfg"
   /var/lib/openhab/config/org/apache/felix/eventadmin/impl/EventAdmin.config:2:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.felix.eventadmin.impl.EventAdmin.cfg"
   /var/lib/openhab/config/org/apache/felix/fileinstall/726218ad-f275-49d2-be3b-c96db7c337ef.config:3:felix.fileinstall.dir="/usr/share/openhab2/addons"
   /var/lib/openhab/config/org/apache/felix/fileinstall/726218ad-f275-49d2-be3b-c96db7c337ef.config:4:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.apache.felix.fileinstall-deploy.cfg"
   /var/lib/openhab/config/org/apache/felix/fileinstall/726218ad-f275-49d2-be3b-c96db7c337ef.config:9:felix.fileinstall.tmpdir="/var/lib/openhab2/tmp/bundles"
   /var/lib/openhab/config/org/jupnp.config:3:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.jupnp.cfg"
   /var/lib/openhab/config/org/openhab/audio.config:3:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.openhab.audio.cfg"
   /var/lib/openhab/config/org/openhab/voice.config:3:felix.fileinstall.filename="file:/var/lib/openhab2/etc/org.openhab.voice.cfg"

Do you have any suggestions on how to fix these issues? Is this something that we can try to fix on the distro side (particulary (1) as this will affect all users using the update script.

[wborn]

Issue 1. is difficult to solve with the current update commands. That's because these commands are currently always executed when doing snapshot/milestone/rc updates of the same version. So if we would add a command to delete the file users would have to reconfigure the installed add-ons with every 3.0.0* update. Some workarounds could be to use a different file/path instead of org/openhab/addons.config with OH3 or having some logic (script/update command) that is only executed once (e.g. by adding some version info to the file).

Issue 2. is a known issue (openhab-distro#860). Users also run into it when switching distro's (.zip -> Debian -> Docker), see also this community topic. Updating the paths in these files should fix it. It's specific to .deb/.rpm packages because for instance the Docker container doesn't use "openhab2" in paths. It might make sense to add this path replacement logic to the backup restore script and use that so data can be more easily restored in different directories.

[BClark09]

Thanks @wborn!

In the case of ${OPENHAB_USERDATA}/config/org/openhab/addons.config, openHAB makes this file itself and adds to this list when you request more add ons, is there no way for openHAB to remove them if they aren't valid? Otherwise, we can add something to the update.lst unless this is too risky?:

REPLACE;paper;;$OPENHAB_USERDATA/config/org/openhab/addons.config
REPLACE;dashboard;;$OPENHAB_USERDATA/config/org/openhab/addons.config
REPLACE;classic;;$OPENHAB_USERDATA/config/org/openhab/addons.config

With issue 2, I'd agree with adding logic to the backup and restore script, we should probably add something to mark the location of $OPENHAB_USERDATA in the backup script before a 2.5.9 release right?

I can solve this for linuxpkg now by adding the lines:

find /var/lib/openhab/config/ -type f -name "*.config" -exec sed -i'.bak' 's|/var/lib/openhab2/|/var/lib/openhab/|g' {} \;
find /var/lib/openhab/config/ -type f -name "*.config" -exec sed -i'.bak' 's|/usr/share/openhab2/|/usr/share/openhab/|g' {} \;

On the first install of OH3, does this seem sensible?

[wborn]

I think the main issue with using the REPLACE update command is that we'd have to add one for each OH1 add-on ID as well (and all other removed UIs). Another issue might be that the feature installer does not skip empty strings (due to ",," leftovers in the file after removing an add-on this way) but we could improve how the Java code handles this.

The feature installer currently doesn't check IDs of available features and keeps retrying add-on installation to recover from various issues like failed online downloads.

I think the find replace makes sense for these packages on the first install and we could use something similar with the backup restore script.

[kaikreuzer]

the feature installer does not skip empty strings The feature installer currently doesn't check IDs of available features

I guess both can and should be fixed. Additionally, I am wondering if we can improve the update mechanism if there was really any change in the configuration and otherwise not re-try failed installations, but rather log an error and have the user do a restart in such a situation.

[kaikreuzer]

I've created https://github.com/openhab/openhab-core/pull/1617.

the feature installer does not skip empty strings

The logic looks as if it does, so I don't think we need to do anything for that.

[BClark09]

I checked and addons.config with a string that looked like: ",,,astro," and it worked for fine.

I'm happy that the linuxpkg side is done, let's enable the build and get people testing 👍

I'll write up how to install and what to expect.

[kaikreuzer]

let's enable the build and get people testing

Done! https://ci.openhab.org/job/openhab-linuxpkg/945/

[ghys]

I get this when trying to install on a system without a previous OH2 install:

$ sudo dpkg -i openhab_3.0.0~S1919-1_all.deb
Selecting previously unselected package openhab.
(Reading database ... 1031228 files and directories currently installed.)
Preparing to unpack openhab_3.0.0~S1919-1_all.deb ...
touch: cannot touch '/etc/openhab2/.copiedToOH3': No such file or directory
dpkg: error processing archive openhab_3.0.0~S1919-1_all.deb (--install):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 openhab_3.0.0~S1919-1_all.deb

After creating /etc/openhab2 it seems to install but there's a lot of weird output:

Preparing to unpack openhab_3.0.0~S1919-1_all.deb ...
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(group) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
Unpacking openhab (3.0.0~S1919-1) ...
Setting up openhab (3.0.0~S1919-1) ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
[openHAB] Please use the following commands to launch openHAB on a system restart.
            sudo /bin/systemctl daemon-reload
            sudo /bin/systemctl enable openhab.service

[openHAB] You can start openHAB manually by executing the command:
            sudo /bin/systemctl start openhab.service

[BClark09]

Thanks @ghys, this is unusual! I'll fix straight away.

[BClark09]

@ghys I fixed the /etc/openhab2 path issue on the first install (and have started a new build) but I couldn't replicate those passwd and group messages. What OS are you running?

[ghys]

Thanks @BClark09!

The "sent invalidate" was an issue with the server itself (it's an Ubuntu instance on Azure, sudo apt-get remove unscd gets rid of these messages).

[ghys]

I switched https://tiny.cc/ohdemo from a manual run in a screen ;) to the .deb package without issues, nice!