Auto provision OH on install

[mstormi]

to be detailed

Related: #1220

In openhabian.conf, specify initialconfig=<some filename> and/or configrepourl=<some Github repo URL>

User can use openhab-cli to create a backup file, copy it off the system, then after flashing a new SD, they can place the config backup file in /boot. If initialconfig points to a valid backup file, openhab-cli restore should be triggered [ mind the space on /boot ]

When the admin provides an OH config repo file, openhabian-setup.sh can git checkout the repo into /etc/openhab so the newly installed OH can start with this config right away (like we do with /opt/openhabian). [ do we need more parameters to provide Git credentials? ]

Eventually implement some special config restore / checkout mechanism (as a parameter to openhabian-config ?) that can be remotely triggered via VPN

Admins can maintain the complete remote instance config in Github, either exclusively or in a cooperative manner together with the user (i.e. you know the game: user screws up things, admin fixes them for him)

[ecdye]

Good idea, certainly worth looking into and implementing.

[mstormi]

Ethan would you be willing to take on this/on #1220 ? I'm busy going after the hotspot one.

[ecdye]

I can try, but I have been a little busy trying to update one of the OH bindings that I use for openHAB3 and get it incorporated into the main branch so it may be a little while before I get to it.

[mstormi]

@ecdye Ethan hope you aren't too busy these days.

Would you consider giving this one (and #1220) a try ? @JAMESBOWLER told me he gave up on it.

[ecdye]

Well, I can probably try, but I'm not great with these things and I think since OH3 the storage has changed because of mainUI. @BClark09 where are all the configuration files for openHAB stored now?

[mstormi]

AFAIK all of the UI definitions (including rules you define in there) are stored in jsondb as XML. The jsondb dir with all its files (except backup/) has to be stored/restored anyway to store things+item that were entered via UI. It has been like that in OH2 already so essentially OH3 does not mean a change. You can reuse or adapt the Git code in openhabian.bash that we currently only use for openhabian itself.

(*) and it was me to write the official docs. Will double-check nonetheless.

PS: there's also the PR @JAMESBOWLER retracted (closed but maybe you can use some of his code). He told me he killed his system with his development attempts so I guess he's not really motivated to go for another shot.

[BClark09]

@BClark09 where are all the configuration files for openHAB stored now?

Generally, if configuration is written by the user then it's in $OPENHAB_CONF (/etc/openhab). If it's written by openHAB then it'll be in $OPENHAB_USERDATA (/var/lib/openhab) the subfolder of which will depend on what addon the configuration is for. These are mostly XML or JSON files.

I'd recommend using the backup zip (which handles all of this) directly. The initialconfig could just be a direct link to download the archive (without being prescriptive to GitHub) or a filepath so that it can be openhab-cli restoreed.

[ecdye]

@mstormi I'm not sure if this is really necessary, shouldn't a user be able to just create a zip and transfer it to a new machine in case of upgrade or use the backup that is included by default if a HW failure occurs?

[mstormi]

It's not always needed but mind you there's various use cases. Not everybody is willing and capable to do it on the CLI. Also check back with #1220 and think of remote installations, i.e. when you operate someone else's setup and don't necessarily have full VPN access to his box. But this way you can maintain his config via Github.

[ecdye]

My other concern is that this involves api token generation and I am not sure that there is a very good way to do this securely using GitHub. It seems like this might be a thing that could better be used in the myopenhab service as a sort of integration on their end.

[mstormi]

You mean to generate the token on the myopenhab servers ? Hmm when and what exactly for would we need a token? In the first place I want to use this to deploy an initial config. I was thinking of something similar to the Tailscale key handling. That's interactively generated offline (on some website yes but Github itself might or might not be enough for this, if not then sure we could look into hosting something on the myopenhab servers) Then it's deployed to openhabian.conf on or after flashing. It could even be built into an image.

[ecdye]

You mean to generate the token on the myopenhab servers ?

I meant the API token generation for GitHub to manage certain repository functions. Additionally git is probably not the best solution for this type of thing anyway.

In the first place I want to use this to deploy an initial config.

Would it maybe be better to allow inclusion of a zip in the initial image and then as part of the setup it attempts to import the openHAB config from the zip?

I was thinking of something similar to the Tailscale key handling. That's interactively generated offline (on some website yes but Github itself might or might not be enough for this, if not then sure we could look into hosting something on the myopenhab servers)

I'm not familiar with the Tailscale key handling process, what exactly are you referring to here?

My thought is that #1220 would be something better implemented by openHAB Cloud where you could manage any openHAB instance from the online cloud service, once the initial config has been done, that way it is kept much simpler and less reliant on things like GitHub. Additionally this allows for other services outside of openHABian to make use of this feature.

[mstormi]

In the first place I want to use this to deploy an initial config.

Would it maybe be better to allow inclusion of a zip in the initial image and then as part of the setup it attempts to import the openHAB config from the zip?

Need to correct/explain myself here. I'd also like to maintain a (text) config that way (overwrite/deploy new rules files for example) without taking OH down of course. That and the fact that some other users successfully do this are the main reason I was thinking to use GitHub for this and benefit from all of its handling features. For example we could tag a set of OH files by date then git checkout to restore the config of a specific date. Or you could have 2 people say a beginner and a consulting expert maintain a config in a cooperative manner like we do with openhabian.

I don't see how myopenhab could do that for us without replicating a lot of git(hub) functionality first.

Regarding Tailscale key handling as I wrote it's generated on a web site then manually entered in openhabian.conf or openhabian-config. We could do the same with a Github key, no ?

[ecdye]

To chase a rabbit trail: my thought regarding myopenhab was that since we are moving towards a UI based config (which is another reason that GH may not be great in the future) you could perhaps remotely manage your config using a new feature in myopenhab. To clarify, when I say remotely manage, I mean myopenhab allows you to interact with the local openhab instance and configure it and make rules etc. from the cloud. Does that make sense?

[mstormi]

I understand what you mean but it is actually way less than what you indicate or hope to be. First, the discussion UI vs text is an ancient one I've been involved in for many years now, it's an ever ongoing one and will not get resolved anytime soon and I believe it will not be decided at all. Text will remain as important and capable as UI and even mandatory for specific usage cases let alone for any type of automated deployment it is and will remain to be the only option. Likewise, there's nothing wrong or outdated or the like to use git/Github to manage configs (you can actually use it to manage all UI-made input, too, which gets stored in files the jsondb directory). Second, myopenhab will not help you with any of the management functions we need. It's essentially just a (selective) HTTP proxy into the local instance's UI. That would make it be(come) at most the same as you can do today with a VPN access, no more. EDIT: as Rich hinted below you cannot modify OH text configs this way. It's interactive only, no automation.

[ecdye]

Well then, based off of what I have seen in this discussion, I think that this tool is still better suited as universal type of solution that one could install on any system to have a complete package to manage a OH system remotely no matter if openHABian or not.

As such I don't really think that this is in the scope of openHABian and should probably be proposed as separate project relating to openHAB that could be created.

@openhab/architecture-council Any thoughts?

[rkoshak]

I mean myopenhab allows you to interact with the local openhab instance and configure it and make rules etc. from the cloud. Does that make sense?

Perhaps I'm missing something or haven't understood the discussion on this thread well enough. But this is already possible through MainUI. If you log into MainUI as user with an admin role, there is very little you cannot configure/edit/create. The exceptions include:

• modify file based configs
• create/modify persistence strategy configs
• fully configure Ephemeris with custom daysets and holidays

That's all I can think of. So what problem is being solved here? I've seen three related but not equivalent things discussed.

1. provisioning a new openHABian install with a set of already created openHAB configs by placing them (backup from openhab-cli?) in the boot partition

2. provisioning and updating an openHAB config via some remote repository (e.g. git) on an ongoing basis

3. provisioning openHAB configs somehow through myopenhab.org

1 seems pretty straight forward and a good idea. 2 Also seems like a good approach for people who are managing and supporting openHAB instances for third parties. Both text based configs and UI created configs are plain text and very suitable for use in a software configuration management system like a git repository (I do all my configs through the UI and use a personal git server to cm them).

3 I don't understand what is being proposed. As I said, you can already interact and manage your openHAB instance through myopenhab, at least concerning what is possible through the UI. For what isn't possible through the UI, I don't see how myopenhab.org is ever going to be suitable for something like that. There is a one-to-one relationship between openHAB instances and myopenhab.org accounts. openHAB doesn't have any way to actually write out to it's own text configs in $OH_CONF. myopenhab.org only proxies openHAB's REST API so it can't really do anything that you can't already do through MainUI. Furthermore, myopenhab.org does not and can not know anything about an openHAB instance until after it's installed and has had the Cloud Connector add-on installed and the myopenhab.org account configured with the UUID and Secret. So I don't see how myopenhab.org could ever be used to auto provision OH on an install. If yet another openHAB service were created, it would have the same chicken and egg problem. You can't use it to configure OH until OH is installed. If it could do that, then really what is being proposed is a generic remote system administration service, of which there are already half a dozen or more already in existence.

Speaking as myself and not as a member of the AC (note all of the above is also speaking as myself) I think there are two issues at play here. One is provisioning OH at install with an already existing config and the other is long term maintenance of that config remotely. Maybe just address the first one now and then figure out the long term part later in a different issue.

[JAMESBOWLER]

A agree with all of you.

I don't really think that this is in the scope of openHABian

provisioning a new openHABian install with a set of already created openHAB configs by placing them (backup from openhab-cli?) in the boot partition

I think this could be implemented, however it is not really a problem for someone that can make a backup of openHAB

provisioning and updating an openHAB config

is that going to be done through UI or text. I access friends and family through SSH with VScode and use text files.

Using the DB files may require openHAB not to be running when updating configs this would get out of sync constantly.

So as per Auto provision OH on install

I think maybe like the restore from backup we could have a Personal openHABian config files update from Github.

Thinking of the non technical person that we encounter they may be able to create a new SD card and stick it in. Then we could put the openHABian menu on a webpage ( I have played with it and did get it working) The non techy could go to the menu option and enter in a link to the personal openHABian config. The script would need to

• [ ] Download config that can be run un-interactively
• [ ] Stop everything from running
• [ ] Reconfigure everything that openHABian can configure
• [ ] restore an Amanda Backup or openHAB backup
• [ ] restart everything and print error when something didn't go to plan with instructions on how to fix the problem.

account configured with the UUID and Secret. You can't use it to configure OH until OH is installed.

Could it be setup to specify the UUID and Secret in the openHABian config?

[mstormi]

On @rkoshak Thanks for your wrap-up, it's pretty much my pov. Just to elaborate a bit on

• provisioning a new openHABian install with a set of already created openHAB configs by placing them (backup from openhab-cli?) in the boot partition

• provisioning and updating an openHAB config via some remote repository (e.g. git) on an ongoing basis

Placing a config tar file in /boot only works for RPis and is often just not feasible. And it is victim to chicken-egg type o' problems. So it's actually a more general incarnation of the 2nd "maintenance" point as that's also supposed to be covering the 1st.

On @JAMESBOWLER

• Download config that can be run un-interactively

The idea of this issue is to only do that in scenarios "initial" or "full restore" ...

• Stop everything from running
• Reconfigure everything that openHABian can configure

... while this is about partial config changes. Small ones most of the time such as to replace a value or a rule at most. These are required to work while OH keeps up & running so it's essentially about retrieving and replacing text files.

• restore an Amanda Backup or openHAB backup

• restart everything and print error when something didn't go to plan with instructions on how to fix the problem.

Those now in turn are out of scope for this issue, let's keep focus.

• Could it be setup to specify the UUID and Secret in the openHABian config?

That's unrelated too, but an interesting thought. Open a separate issue for that please.

[ecdye]

@rkoshak @mstormi Thanks for the further insight.

I don't think I am ready to implement this at this time, I feel that it is (especially in my case) a little difficult to justify spend that much time on a feature that has no impact on me. I might revisit this in the future but for now I'm going to open it up to anyone who would like to give it a shot.

To be clear, I do not think that this is a bad idea, just I am not willing to put in the time and effort to implement and test a solution at the current time. I would be more than happy to review and give pointers to anyone who would like to implement this.

[mstormi]

Fixed-By: #1515

the initialconfig could just be a direct link to download the archive (without being prescriptive to GitHub) or a filepath so that > it can be openhab-cli restoreed.

right like that, no GitHub involved for the time being. Anything git should be implemented in #1220