migration path / namespace migration

[mstormi]

The overall namespace change from org.eclipse.smarthome to org.openhab needs to be reflected in JSONDB and possibly more locations. This means to a) change some config file names and b) change namespaces inside these. See https://community.openhab.org/t/oh2-things-gone-missing-in-oh3/109094/2

I'd think the best place to migrate a config is the restore script distributions/openhab/src/main/resources/bin/restore, wdyt ?

[kaikreuzer]

Could you be more specific, please? This should all already be covered in https://github.com/openhab/openhab-distro/pull/1030 and https://github.com/openhab/openhab-distro/pull/1057.

[mstormi]

Short of a migration tool, I installed 3.0M3 package from scratch and copied my OH2 config file over next (I actually used openhab-cli restore albeit fixed the path - see also #1190) The JSONDB file was of course still named org.eclipse.smarthome.yaddayadda and also contained several strings with the old namespace, too. How is that supposed to be changed if not by the restore tool ?

[kaikreuzer]

Copying openHAB 2.x config files onto a 3.x installation is simply not a supported use case. Only the mechanisms that make use of the update.lst file are supported, because that's the file that contains all relevant information on how to migrate user data. If you think it would be valuable to have a script that does a data migration from version X to Y, this can theoretically be implemented - it's just that somebody would need to find the time to create and maintain it...

[mstormi]

but isn't this what any user would do by intuition ? I don't even know what the proper migration path would be i.e. what the "mechanisms that make use of the update.lst file" are so expect this to be a top scorer on the list of OH3 issues raised by users. Sure a full X-to-Y data migration script is desirable but probably out of reach. I'm still chewing on doing that for the openHABian (i.e. non-openHAB) part.

[kaikreuzer]

but isn't this what any user would do by intuition ?

Hopefully not, because it was never ever supported to update your openHAB installation that way.

I don't even know what the proper migration path would be

Well, clearly the ones that are the documented update mechanisms:

https://www.openhab.org/docs/installation/linux.html#upgrade https://www.openhab.org/docs/installation/linux.html#apt-based-systems-2

To be honest, I am not sure how well they work for a 2.x to 3.x switch, especially as the linux package names have changed (@BClark09 is the expert here). I also don't see anybody working on a smooth migration path with a good tutorial for users - so that's definitely something to focus on and I guess help is required. I myself am unable to look into it as there's far too much other urgent stuff on my plate that needs to be finished :-/ . But that's why my assumption is that we should get the existing (and officially supported) update mechanisms working as expected instead of trying to introduce yet another one.

[mstormi]

Remember we have directory name changes openhab2->openhab this time, more than one. So files cannot be migrated in place, there's always a need to copy in one way or another. Hence my proposal to implement it in the restore script. This has been giving me quite a lot of headache already in openHABian and even resulted in a new openHAB3 branch and to be frank I don't know either how well that will work.

[kaikreuzer]

Remember we have directory name changes openhab2->openhab

That's what I meant with Linux package rename and hope that @BClark09 can chime in. If by this we do not have any migration path for Linux package users in place, I agree that some solution must be found (possible something similar to the manual "update.sh" script, just without actually downloading the zip distro).

[BClark09]

Hi all, I've been struggling with time recently due to everything going on in the world but I'll do my best to give information.

The Linux packages are set up to use the update.lst file on each update to a new version. It is also setup to do this once during a change from openHAB 2.X to 3.X when migrating using the instructions in my post. If there is an issue with migration or upgrading, it would be because update.lst in the distro needs updating, or the function definitions that linuxpkg uses need changing.

From my understanding @mstormi, you didn't use linuxpkg to update, but you tried starting from a fresh installation and using the distro's backup and restore scripts? With regards to your original forum post, the apt installation would have done those steps for you. I don't think that backup/restore is setup to do this, but perhaps they should be: the backup's store version information so the restore script can be modified to include the steps listed in the update.lst file. In additions other issues with the backup/restore scripts need to be addressed.

[mstormi]

We must not rely on people to be aware of this part of the docs (even I missed it) and benjy's post. The update docs are at the very bottom of the lengthy linux page up to where noone's reading, unclear that they apply to OH3 in the first place, give /opt/openhab as the installdir example (while on most systems including openHABian it's /usr/share/openhab/) and say to run "update" which is just a fake script telling you to use your package manager mechanism. @Confectrician could you take care of a prominent placement of OH3 install/migration docs ? For the time being, I suggest you create a tutorial based on the instructions in benjy's post (link see last comment). Post the link here, we all can then make use of the usual edit mechanism (that creates PRs against openhabian-docs) so changes are tracked, too.

However now the IMHO worst part remains .... while update/update.lst change the JSONDB file name, I have not seen a file contents namechange handled anywhere so I guess names don't get replaced even if I follow the instructions in benjy's post. @BClark09 can you extend postinst to also replace all occurrences of org.eclipse.smarthome to org.openhab ? This should be applied at least to all files in $USERDATA/jsondb and to $USERDATA/etc/org.ops4j.pax.logging.cfg as well. And likely more files, will post here if I stumble across others. [ in order not to forget this, at least in the log definition file we need a replacement org.eclipse.smarthome.model -> org.openhab.core.model (note the "core"), too, so log settings are kept across migrations. Think UX - log output missing will be a thing people get mad at, in particular because of the increased need for debugging on migrating. ]

Either way, @BClark09 I agree we should add those replacements to the backup/restore scripts as well. Can you also trigger the postinst script at the end of a restore ? Do all package managers make use of this ? Duplicating code would only be 2nd best solution.

[kaikreuzer]

We must not rely on people to be aware of this part of the docs

We cannot support any path that people that don't read documentation THINK how an upgrade might work. Try copying your Windows XP registry files over to a Windows 10 installation and complain at Microsoft, why it isn't working as you expect...

We always documented the upgrade process in our release notes and we will do the same for the 3.0 release. We must expect people to read it or at least be allowed to point them at it, if they didn't read it.

However now the IMHO worst part remains .... while update/update.lst change the JSONDB file name, I have not seen a file contents namechange handled anywhere

This is done by the REPLACE parts of https://github.com/openhab/openhab-distro/pull/1057. Please test the migration and report any parts that do NOT work.

[mstormi]

We cannot support any path that people that don't read documentation THINK how an upgrade might work.

Well I, too, am telling users at least twice a day to RT(F)M, but your M$ comparison is a little too much IMHO. I'd still advocate to think just a little less like a dev and to take a little more of the user perspective. Most users will try to migrate the way they think is intuitive anyway, regardless of whether's that's documented. Even more so as the current upgrade documentation isn't intuitive, hard to find, outdated (written for OH2) and possibly even wrong.

Mind you that it also requires people to upgrade in place and they don't have the option of reverting back if something goes wrong (short of a backup/restore) because there is no data downgrade available. There's likely users to refrain from upgrading because of this in the first place. Only a few will have secondary boxes available, and even those would have to prepare them with 2.5 and their config before (!) they install OH3 to be able to get them upgraded during OH3 install. OpenHABian users will possibly seize the opportunity and install a new box so no in-place upgrade either. And documented or not, us being more proactive here will also result in less efforts we need to spend on support later on, let alone a better overall user experience and reputation.

So I think we should get the tutorial ready (waiting for @Confectrician to make a start) and at least add postinst execution to the restore script (waiting @BClark09) so we can promote openhab-cli backup/restore as a supported migration path (maybe even as the preferred one?). Will also test and update here if I find issues.

[mstormi]

Hmm, the current upgrade instructions Kai you linked to are pretty useless for OH3: The "update" script just prints a message to have it handle the pkg manager. The apt/yum/ ... upgrade instructions (such as apt install openhab=[version]) don't apply because the package name changed. Using that new one (apt install openhab), a simple repository based install works (I moved on with Ben's instructions at this point). But in my first attempt it did not migrate the openhab2 config because while the config files existed, I had uninstalled the package. Note that's what openHABian does today, too: apt remove openhab2; apt install openhab. That left openhab2 in a state

[21:24:19] root@openHABianDevice:/etc/openhab# dpkg -s openhab2
Package: openhab2
Status: deinstall ok config-files

I moved on and installed openhab2 again, then tried another "migration" install run of the OH3 package:

[21:55:07] root@openHABianDevice:/etc/openhab# apt install openhab
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  default-jre | java11-runtime | java11-runtime-headless
The following packages will be REMOVED:
  openhab2 openhab2-addons

So if I hadn't removed OH2 myself before it would be removed now (I even installed it again for a test - yes it gets removed)

And like the first time, no config was copied/migrated.

Maybe I overlooked something but I'd think I followed the instructions, and they don't work. Well OH3 gets installed. But the OH2 config ain't migrated.

@BClark09 I wouldn't think it's truely mandatory to have the OH2 binaries installed, is it ? So can you change your "openhab2 is installed detection" so it's ok with the shown state and still grabs the config off there?

Another thing (shall I open a new issue?) is

$ apt-get install --allow-downgrades --yes openhab=3.0.0~S2038-1 openhab-addons=3.0.0~S2038-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  openhab openhab-addons
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 254 MB of archives.
After this operation, 3,072 B of additional disk space will be used.
Get:1 https://openhab.jfrog.io/openhab/openhab-linuxpkg unstable/main armhf openhab all 3.0.0~S2038-1 [82.2 MB]
Get:2 https://openhab.jfrog.io/openhab/openhab-linuxpkg unstable/main armhf openhab-addons all 3.0.0~S2038-1 [172 MB]
Fetched 254 MB in 3min 3s (1,390 kB/s)
apt-listchanges: Reading changelogs...
(Reading database ... 143490 files and directories currently installed.)
Preparing to unpack .../openhab_3.0.0~S2038-1_all.deb ...
Unpacking openhab (3.0.0~S2038-1) over (3.0.0~S2037-1) ...
Preparing to unpack .../openhab-addons_3.0.0~S2038-1_all.deb ...
Unpacking openhab-addons (3.0.0~S2038-1) over (3.0.0~S2037-1) ...
Setting up openhab (3.0.0~S2038-1) ...

[openHAB] Performing post-update tasks for version 3.0.0:
  Moving:   From /var/lib/openhab/config/org/eclipse/smarthome/i18n.config to /var/lib/openhab/config/org/openhab/i18n.config
  Replacing: String org.eclipse.smarthome.i18n to org.openhab.i18n in file /var/lib/openhab/config/org/openhab/i18n.config
  Moving:   From /var/lib/openhab/config/org/eclipse/smarthome/persistence.config to /var/lib/openhab/config/org/openhab/persistence.config
  Replacing: String org.eclipse.smarthome.persistence to org.openhab.persistence in file /var/lib/openhab/config/org/openhab/persistence.config
  Moving:   From /var/lib/openhab/config/org/eclipse/smarthome/audio.config to /var/lib/openhab/config/org/openhab/audio.config
  Replacing: String org.eclipse.smarthome.audio to org.openhab.audio in file /var/lib/openhab/config/org/openhab/audio.config
  Moving:   From /var/lib/openhab/config/org/eclipse/smarthome/voice.config to /var/lib/openhab/config/org/openhab/voice.config
  Replacing: String org.eclipse.smarthome.voice to org.openhab.voice in file /var/lib/openhab/config/org/openhab/voice.config
  Moving:   From /var/lib/openhab/config/org/eclipse/smarthome/links.config to /var/lib/openhab/config/org/openhab/links.config
  Replacing: String org.eclipse.smarthome.links to org.openhab.links in file /var/lib/openhab/config/org/openhab/links.config
  Replacing: String org.eclipse.smarthome.folder to org.openhab.folder in file /var/lib/openhab/config/org/openhab/folder.config
  Moving:   From /var/lib/openhab/config/org/eclipse/smarthome/threadpool.config to /var/lib/openhab/config/org/openhab/threadpool.config
  Replacing: String org.eclipse.smarthome.threadpool to org.openhab.threadpool in file /var/lib/openhab/config/org/openhab/threadpool.config
  Replacing: String org.eclipse.smarthome.autoupdate to org.openhab.autoupdate in file /var/lib/openhab/config/org/openhab/autoupdate.config
  Replacing: String org.eclipse.smarthome.core to org.openhab.core in file /var/lib/openhab/jsondb/org.openhab.core.config.discovery.DiscoveryResult.json
  Replacing: String org.eclipse.smarthome to org.openhab.core in file /var/lib/openhab/jsondb/org.openhab.core.config.discovery.DiscoveryResult.json
  Replacing: String org.eclipse.smarthome.core to org.openhab.core in file /var/lib/openhab/jsondb/org.openhab.core.thing.Thing.json
  Replacing: String org.eclipse.smarthome to org.openhab.core in file /var/lib/openhab/jsondb/org.openhab.core.thing.Thing.json
  Deleting Directory: /var/lib/openhab/config/org/eclipse/smarthome
  Deleting Directory: /var/lib/openhab/config/com/eclipsesource

dpkg: error processing package openhab (--configure):
 installed openhab package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of openhab-addons:
 openhab-addons depends on openhab; however:
  Package openhab is not configured yet.

dpkg: error processing package openhab-addons (--configure):
 dependency problems - leaving unconfigured
Processing triggers for systemd (241-7~deb10u4+rpi1) ...
Errors were encountered while processing:
 openhab
 openhab-addons

Now strange enough, when I repeat it works out:

[23:41:57] root@devpi:/etc/openhab/sitemaps# apt install openhab
Reading package lists... Done
Building dependency tree
Reading state information... Done
openhab is already the newest version (3.0.0~S2038-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up openhab (3.0.0~S2038-1) ...

[openHAB] openHAB was not running so will not start after upgrade.
[openHAB] Please use the command:
            sudo /bin/systemctl start openhab.service

Setting up openhab-addons (3.0.0~S2038-1) ...
Updating FireMotD available updates count ...
[23:45:15] root@devpi:/etc/openhab/sitemaps#

[mstormi]

@BClark09 please see https://github.com/openhab/openhab-distro/pull/1199 for config migration on restore Drafted that only - still looking for a more clever way to reuse the code from the postinst script ? AFAIK unlike update.lst it's not permanently stored on the system so we cannot reuse it (or split that maybe to have a permanent part remain on the system after install?)

[BClark09]

Maybe I overlooked something but I'd think I followed the instructions, and they don't work. Well OH3 gets installed. But the OH2 config ain't migrated.

@BClark09 I wouldn't think it's truely mandatory to have the OH2 binaries installed, is it ?

You removed openhab2 separately before installing openhab. My instructions say to do this in one command:

sudo apt install openhab

This will prepare openHAB 3.x with information from openHAB 2, remove openHAB 2's binaries, and then complete openHAB's 3s migration. The order is important here.

So can you change your "openhab2 is installed detection" so it's ok with the shown state and still grabs the config off there?

No, we have no easy way of telling how old/valid this uninstalled openhab package is. Having dpkg remove openhab2 for you guarentees that you only migrate with a valid version of openHAB 2 and aligns the behaviour with yum, docker and the standalone build of openHAB.

[mstormi]

So can you change your "openhab2 is installed detection" so it's ok with the shown state and still grabs the config off there?

No, we have no easy way of telling how old/valid this uninstalled openhab package is. Having dpkg remove openhab2 for you guarentees that you only migrate with a valid version of openHAB 2 and aligns the behaviour with yum, docker and the standalone build of openHAB.

But there can only be a single version of the OH2 config, can't it ? That one in the openhab2 dirs. So that's the one to get migrated for the user. If it's too old to work in OH3 then so what ? A standard disclaimer to re-do everything from scratch if it doesn't work after a migration will apply anyway won't it.

Also, dpkg -s will show Status: deinstall ok config-files if you have removed (but not purged) the package like I did in my attempts. So while the package is gone, config is still there and can be migrated. So why not do that ?

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/docker-2-5-12-error-core-karaf-internal-featureinstaller-failed-installing/121517/18

[kaikreuzer]

I'd say this can be closed now that openHAB 3 is out since 6 months.