Create a CLI to import existing and future YAML configs into .storage entities

Context

In #399, it was proposed to bring YAML configurations back arguing on a list of broken user flows. The alternative proposed by @Jc2k is to create a CLI to do that import.

Note: While @Jc2k does not believe this should be done by Home-Assistant architecture, I am opening this on this forum for 3 reasons:

While @Jc2k proposed that this is done by the Community, I am requesting a discussion on whether this should and could be done by Home-Assistant core (as such this is the right place).
Regardless of whether this done by the Community or Core, this is used as an argument to avoid changing ADR-0010. I think discussions are still needed to clarify if this is really solving the issues and whether it would be viable without further changes.

Proposal

One of the things that was touched on in #143 and #370 was the possibility of using the very same API's that the front end uses and provide a CLI tool that could apply yaml configuration via those API's. In those tickets i suggested something that worked a bit like kubernetes manifests and kubectl apply.

It seems like the community could even build a tool like this without needing a large ADR to pass, and many if not all of the needed API's must exist for the front end to make these devices work in the first place.

This would allow you to put your manifests in Git, to back them up, to share them. Different sections could be restored individually. The manifests could contain comments. You could run a sed over all your manifests and apply them with your cli tool to do bulk updates.

Consequences

There are some compelling advantages to this CLI approach too:

Most of the API's needed already exist, otherwise the frontend wouldn't work
If the API's end up needed to be richer or better, this will ultimately benefit the front end as well
It let's people who want the UI and people who want a CLI based approach collaborate
If it's done well then you don't need to update 100's of integration to get decent coverage. You just need to add nice API's for working with ConfigFlow and OptionsFlow.

Bringing relevant discussions on #399 here:

Reply by @nitobuendia :

Hi @Jc2k, thanks for your answer and time.

First of all, my proposal here is to simply approve support YAML configuration. I am not trying to recommend in what way it should be done. For that we can open other issues. CLI is one option (I will reply to it soon). This one could be another one.

Config flow originally existed for devices that cannot be easily configured in yaml. Devices that aren't static or are awkward to configure manually. For example, where you need to perform an oauth flow. Or home homekit_controller needs to a 6 step cryptographic exchange. How do you see that working in YAML?

UI is a good way to generate those. Once generated, they can be stored back in YAML. See this proposal.

I think this approach will be annoying for end users when a maintainer doesn't have capacity or willingness to implement yaml support.

It is annoying. This is why we are complaining. My position is that having some is better than having none. Also, if the maintainer doesn't have capacity, maybe other contributors will. That's how Open Source is meant to work. Forbidding it will still annoy the end users, and also some maintainers who may want to support it.

I think it will be distressing for volunteers when a user gets upset on a ticket (or angry and aggressive and threatening).

People should be allowed to file feature requests. People threatening should be blocked. We should not take product decisions based on threats. I think the position here was that by making it forbidden officially people cannot threat. 1) They may threat for other reasons and 2) this position still states that YAML is not a requirement.

And it might be confusing when triaging tickets - is the config flow in play or is this an integration that implements yaml as well?

How was dealt with until now? One simple idea: can that be added to the bug request template? That's how other confusions are handled (like which installation method, etc).

In addition, not having YAML will also harm troubleshooting. You can read about it on this section.

This would allow you to put your manifests in Git, to back them up, to share them. Different sections could be restored individually. The manifests could contain comments. You could run a sed over all your manifests and apply them with your cli tool to do bulk updates.

This sounds like a fair solution. If I can keep my configuration.yaml and a system takes care of translating it to the format Home-Assistant needs, that's what we are asking.

As stated before, I am not proposing using method A or B, I am proposing to allow YAML to exist for all integrations. The HOW can be discussed later. CLI might be a good solution.

However, I do not think all would be solved:

If CLI needs to be manually run. All configurations will still eventually break. If you acknowledge that there's a way to translate configuration.yaml into entries via the same APIs: why drop YAML support? Make this part of core, run it at start up and translate configuration.yaml into .storage.
Reduced documentation for users and developers. If YAML is not officially supported, what's the schema of the YAML that should be followed? How does the CLI know what to translate into what? In other words, how does the user know what to put in all parameters?

Reply by @Jc2k:

The idea is that the CLI is manually run, and is like using the UI but on the command line. Remember this is using the same REST API as the UI. It is in fact a UI. This is interesting because:

It potentially allows hybrid cases where the majority of the manifest is static config, but it is mixed together with dynamic data like a pairing code before being sent to HA.
Changes can be applied without powercycling HA (like whe using config flow UI)
You could apply your config changes remotely. Anywhere where you could use the web frontend. This could be useful for e.g. power users that use CD (i used to before i embraced the UI). You can edit the manifests locally on your laptop and push them. No need for remote drives or ssh or the VScode integration.
Only changes that pass validation end up in your config. It is much harder to apply a broken config.
We can potentially read yaml manifests back out of a system that didn't use them initially
It can start off as a commuity project and prove its useful without any HA developers doing anything

I don't agree with (2). It might be that people don't write good enough documentation any more but there isn't a technical reason for bringing back yaml. Maybe there needs to be more requirements when writing docs?

Anyway, if there was a community project or official project to make a yaml tool it would obviously need documentation. That documentation needs writing for either approach.

In the CLI case remember that there is a web UI for this. It's already programmatic. What's to stop the CLI leveraging it too? E.g. It's possible that we can use the (translated) strings.json and the validation data already in the configflow forms to provide a schema.

Creating and maintaing such a CLI is out of scope for the Home Assistant project. Feel free to create and maintain one yourself.

Hi @Jc2k

The idea is that the CLI is manually run, and is like using the UI but on the command lie. Remember this is using the same REST API as the UI. It is in fact a UI.

I could imagine something within the Configuration UI to "Import Configuration". It could make sense.

However, I think there are still a few loopholes to solve:

If this must be run in the UI, then what happens when the UI breaks like the case of nmap that I reported?
How do you do partial imports / backups?
Why can't this be run in the backend using the underlying Python code and therefore be able to run as a script as well (in addition to the UI)?

With UI (unrelated to CLI):

How will people report their configurations to be able to reproduce bugs?

I don't agree with (2). It might be that people don't write good enough documentation any more but there isn't a technical reason for bringing back yaml. Maybe there needs to be more requirements when writing docs?

That's another way to look at it. However, that's what UI is bringing. For example, if I look at PS4 component, it doesn't mention that the data required is Region, PIN, IP and token (and name, and similar). It also doesn't explain how this data is stored in .storage. All in all, it would be harder for anyone who wants to contribute.

It potentially allows hybrid cases where the majority of the manifest is static config, but it is mixed together with dynamic data like a pairing code before being sent to HA.

That can also be solved with other YAML mechanisms. See #244

Changes can be applied without powercycling HA (like whe using config flow UI)

Why is this not possible with the current system? I am aware of the limitation but not the root cause. Maybe that's something that cna be improved.

You could apply your config changes remotely. Anywhere where you could use the web frontend. This could be useful for e.g. power users that use CD (i used to before i embraced the UI). You can edit the manifests locally on your laptop and push them. No need for remote drives or ssh or the VScode integration.

This implies access from external network. SSH can provide also remote access. Not sure why that'd be a negative point.

Only changes that pass validation end up in your config. It is much harder to apply a broken config.

Good benefit. I like it.

How would that work? That requires a schema being defined, right? The schemas do not seem clearly defined on UI config flow, or am I missing something?

In any case, I'd say it's a nice have. If you're tinkering, breaking the system is expected. If you do not want to break it, stick with UI.

We can potentially read yaml manifests back out of a system that didn't use them initially

This would be great. This is possible today regardless. See the same approach I took in #400 but using the stored data rather than the "to-be-stored" data.

It can start off as a commuity project and prove its useful without any HA developers doing anything

I don't think so.

How do you expect that to happen if the data in .storage is not documented? Should someone just guess for all integrations which are the required data and configurations? What about those UI APIs? Are they all well documented?

In addition, the cost of maintenance of a centralized (and not integrated) system that adapts YAML for all components is much higher than either a decentralized or an integrated one.

As there's no defined and documented schemas, a contributor may change the structure of the .storage entities and break such solution.
As it's not integrated, there will be no notification.
As the system is centralized, the contributors need to create the YAML approach for all the integrations and keeping track would be hard.

I really do not think this would be viable, but I feel it's a good mechanism to leverage if what you want is to prove that no one is interested in such thing as the entry cost is just way too high. Pretty much the same reason for which there's not a good fork of Home-Assistant yet. A fork would be an easier and more stable way to solve it.

Hi @balloob

Not sure how you could read the whole proposal in less than a minute, but I am not too clear on why this would be outside of the scope of Home-Assistant? Home-Assistant has broken multiple user flows with ADR-0010. CLI is one potentials solution that core team could bring to minimize the impact of those user flows. As such, my proposal is that this is natively supported by Home-Assistant, as I mentioned on the original message.

We are a limited set of people that work on Home Assistant. We need to focus or we can't achieve anything. We have over 400 open PRs and here is a proposal to increase our scope of work, while we even can't handle the current load. We're not interested in it. Do it yourself and maintain it yourself.

You're very much into writing long ass proposals yet you have never taken the time to contribute as much as a single line of code to Home Assistant. You keep making it look so easy and gloss completely over the fact that we have to maintain it.

So no, we're not doing this. Or any of your other proposal.s

Hi @balloob , interesting position.

You're very much into writing long ass proposals

The reason for which I invest the time to think, write, review and then post these proposals are (source) is: Collaboration, Inclusiveness and Community.

"Only by including diverse perspectives in our conversations can we be certain we've identified the best ideas, and decision-makers continually seek those perspectives:
"Shared values guide decision making, and community goals supersede individual interests and agendas."

You may not value my time in bringing them, or these values, but it is how Open Source works.

yet you have never taken the time to contribute as much as a single line of code to Home Assistant. You keep making it look so easy and gloss completely over the fact that we have to maintain it.

There might be employment contractual reasons for this. I would love to contribute more ;)

What I have done is what I can: create custom components (e.g. 1, 2), and put the thought effort in designing creating experiments trying to find broken flows that Home-Assistant has.

This being said, I do not think opinions of others are less valuable. Anyone's time invested should be considered valuable. If you do not agree, I feel very sorry for the project.

So no, we're not doing this. Or any of your other proposal.s

This is not very nice words, or Open-Source driven.

You are shutting down all my (future) proposals without knowing whether they will be good for Home-Assistant, the users or the Community; just based on who proposes it. It doesn't feel right.

<joke>In that case, could I kindly propose to remove YAML and make all components UI only? Maybe it was all about proposing the opposite.</joke>

Finally, taking the opportunity to share the GitHub Community Guidelines, which I feel may be relevant to the tone of this conversation:

Be welcoming and open-minded - Other collaborators may not have the same experience level or background as you, but that doesn't mean they don't have good ideas to contribute. We encourage you to be welcoming to new collaborators and those just getting started.
Communicate with empathy - Disagreements or differences of opinion are a fact of life. Being part of a community means interacting with people from a variety of backgrounds and perspectives, many of which may not be your own. If you disagree with someone, try to understand and share their feelings before you address them. This will promote a respectful and friendly atmosphere where people feel comfortable asking questions, participating in discussions, and making contributions.

The entire point of the CLI idea is that it doesn’t burden the HA team. It creates no new work for them because they are overstretched as it is. It is something the community could work on. I am unhappy that despite saying I didn’t want or think a HA ticket was needed or appropriate you went ahead and tried to create one on my behalf, not only that but suggested that the core team should make it. This is the exact opposite of one of the main goals of the idea - to empower the yaml community without any burden on upstream.

I’m sure you meant well, but it’s still rude. Despite making the ticket on behalf and copying my comments into here my thoughts around this belonging as a arch ticket haven’t changed. I won’t be debating you about the technical issues of the idea on this ticket (it’s closed now anyway). That’s not how open source works.

Hi @Jc2k

I don't see why this cannot be brought as a feature request (even if that was not your original intention, I did clarify that). I still disagree with your perspective. I also do not think this would work and solve the current user flows, nor can be implemented by the community.

Nonetheless, apologies if you found this rude. As you said, it was not the intention. I do am disappointed that you are not open to discussing, unfortunately.

Write it and put it in your own repo.

Write it and put it in your own repo.

Hi @nickrout -- Not sure if that was addressed to me, but I'll assume so given that I am the Author of the issue.

If you read my replies, you will see that I do not think that CLI solves all the problems; or I do think it's a feasible Community project. (Do note that I am not just talking about technically feasibility, which I do think it mostly is). As such, I do not see value in "putting it on my own repo". I also do not think it's perfect if it's part of the Core, but at least I think it's more viable. In any case, my point here was to showcase that it's not viable and that user flows (#399) are still broken with no viable solution. That's why I open this for debate.

As for all those "do it yourself comments" (there has been a few), I would love to address them at once.

First, I am fortunate to be a developer. As such, I have solved my own problems through custom components. I have also shared with the Community so people can use them. The only component that I use that currently does not have YAML support is PS4.

If you are in the same situation, you can get it here. I do not create custom components for components that I do not own, because it would be really hard for me to troubleshoot and confirm that it all works.

This being said, others are welcome to contribute to the repository. I also explained how this can be done and intend to improve documentation.

Hence, I do feel I am doing my part; within my own limitations, restrictions and capabilities. (You may want to have a read at the "Be welcoming" section of GitHub).

Second, not everyone is as fortunate to be able to code their own CLIs or custom components. Thinking that "do it yourself" is viable, it's not fair to the users. While Home-Assistant is free and open-source, the people using this are still Users. Users who have trusted this Project so much that they have put it in their homes. I personally think it would be stupid to break that trust just because "they can do it themselves".

In short, I am not raising this just for myself, I am lucky to be patch it enough that it works for me. I am raising this for all the people who are not developers and are indeed stuck with broken user flows described in #399. Hence, why I spent the time to inform myself, summarize opinions and bring them for debate. This is still valuable and while it's not code, I do not see the harm in contributing this way; given my circumstances (which, of course, I do not expect you to understand).

That being said, this is going off-topic, which is not a good practice or aligned with the community guidelines. Let's stick with the CLI request.

Thanks for your contribution, Nito

You're very much into writing long ass proposals yet you have never taken the time to contribute as much as a single line of code to Home Assistant.

I would kill to get @nitobuendia's sort of detailed and relatively open-minded proposal on any software I work on, instead of the rambling two-paragraph demands from users and non-technical managers that often happen. I know Home Assistant has received a great many of those latter type recently, just from ADR-0010 alone.

I can't blame the core maintainers for reacting poorly to a wave of vitriol, but at the same time I don't understand the outright denial of any community attempt to understand or influence such a significant design change. Here (and in #399) are well-fleshed-out proposals—to the extent it is possible to do so with publicly-known information about the project and its roadmap—clearly articulating community concerns and attempting a compromise solution.

You keep making it look so easy and gloss completely over the fact that we have to maintain it.

And #399 is attempting to articulate why this is an important feature to the community that is worth continued maintenance, while this issue is proposing one possible approach which would somewhat decrease the maintenance load on the core team while meeting the needs of a large or at least vocal part of the userbase. It is not a demand to simply reverse the decisions of ADR-0010, but an attempt to find an alternative solution that will make both the core team and the larger community at least somewhat happy.

So no, we're not doing this. Or any of your other proposal.s

This is not an appropriate response to an invested contributor to an open-source project (and @nitobuendia is a contributor, despite your attempts to disparage him based on LoC metrics), and I feel reflects a growing resentment and distrust from the core team toward other contributors and non-contributor community members.

I have to ask, and I realize this question is more likely to get a ban than a written response: is this an open-source, community-driven project, or is it an internally controlled project of @balloob/Nabu Casa using the availability of its source code to solicit third-party labor?

home-assistant / architecture