search

home-assistant / architecture

Repo to discuss Home Assistant architecture
317 stars 99 forks source link

Integration quality scale #20

Closed balloob closed 6 years ago

balloob commented 6 years ago

I want to introduce a scale to judge the quality of each integration. An integration is a combination of component/platform for a service or device.

This scale will guide developers in how to improve their integrations and will guide users in how "good" an integration is. (I don't like the word quality but can't come up with a better name atm)

Each section extends the requirements from the previous one.

Basic

Medium

Expert

Integrations that reach expert level will require the code owner approval for each PR.

Landrash commented 6 years ago

Suggest we call it Component Integration Scale. Removes the word quality but still keeps the self explained name.

Some adaptation should be made for "native" components to indicate to indicate that they are exempt (If they are) from this.

balloob commented 6 years ago

I would expect this scale to only apply to integrations of third party services/devices. It would not apply to something like the automation component.

amelchio commented 6 years ago

Looks good to me. How do you plan to use this categorization?

I am thinking of a word like "maturity". Also, "full" rather than "expert". I expect the requirements to change over time so using an absolute term is not as bad as with USB full-speed.

balloob commented 6 years ago

I like maturity, although what I like about the word "quality" is that it is clear what it means to anyone seeing a clear . And "full" definitely sounds better than "expert"

I think that this categorization will be used in various ways:

NovapaX commented 6 years ago

I think this mixes two things:

Both are important, but not necessarily correlated Bad code can still give a proper user experience (until it break because of missing tests...). Expert level code, without auto-discovery or config_entries and a huge complex config-schema will still give mediocre user-experience imo.

balloob commented 6 years ago

So I think that having these be part of the quality requirements is very important.

NovapaX commented 6 years ago

Yes, that is true. Both should be part of the score, because both are important. But a component not being very advanced in code should not get a lower value if it does provide a very good UX.

So I think you should start with the maximum score/level. And you substract points for bad code, not having tests when it should have, not having a nice setup experience (complex config) or other not-nice things. That way you don't punish simple components for being simple. You see what I mean?

But anyway, this is a great idea to give users some feedback on what to expect, and devs on what to improve.

kellerza commented 6 years ago

The metrics proposed are measurable. Good UX is subjective, even more so since we might not use or even have access to certain devices.

A "Level of integration" badge in the documentation could be a nice addition, with Bronze/Silver/Gold badges. And maybe a native HASS blue badge for things like automation

Mature can also be open for interpretation, since complex code might not mean bug free (tests do help, but integrations can be temperamental!)

OverloadUT commented 6 years ago

I'm late here, but I love this idea. I think it would be similar to the "IoT class" we have listed on a lot of platforms. I actually use that IoT Class rating as one of my main decision makers when buying a new product. Pairing the unicorn "local push" badge with the "expert quality integration" badge would be a great way for users to know that a product they are thinking about buying has best-in-class integration with Hass.

(Edit: I bring up the IoT Class badge because I think it might make sense to display both things in the same place)

stephenmg12 commented 6 years ago

It would also be helpful to know how stable an integration is. Is it an official documented API or did we reverse engineer the app? An example would be Lutron breaking things.

OverloadUT commented 6 years ago

That gets a tad complicated because officially we're using third party libraries to do all of the I/O.

I do like the idea though. It definitely supports my "decide which products to buy" use-case.

balloob commented 6 years ago

I went ahead and opened a PR for an initial version. It's based on the proposal in this issue, the comments and recently introduced functionality.

I switched to using silver, gold and platinum. The reason I don't do bronze - gold is because the 2nd tier is already a very good experience.

Preview Integration Quality Scale

awarecan commented 6 years ago

Platinum badge looks cheaper than gold one 😄

How about 🏆💎👑

awarecan commented 6 years ago

Besides a quality scale, I also suggest that we have several other badges/labels we can assign to a component

We can also have some badges for single featue to give devs a little accomplishment, such as

dgomes commented 6 years ago

I think @awarecan idea is better then the scale.

We could just have several badges and the scale becomes dynamic as a result of the number of badges collected by each component.

The badge approach has the benefit that developers can tackle badges one at a time.

awarecan commented 6 years ago

So it will like a matrix

Scale Badge
No rank 🏛️official, 📜api, 🎩 black hat, ✔️Apache 2 license compatible, 🚀experimental
No score basic, configuraion.yaml
🥈 scan_interval, paltform_not_ready, auth_handle, internet_outage, device_outage, unavailable, unique_id
🥇 config_ui, config_entry, device_info, discoverable, l10n, tests, code_owner
💎 parallel_updates, config_unload, async_all, aiohttp_session
fabaff commented 6 years ago

To get started we should keep it simple. Meaning go with three grades and refine it then or make a scale according to the fulfilled requirements.

It took me hours to get the IoT classes for the available platforms/components back then. And now we have at least doubled the amount of integrations. It would already require pretty much a review of every single integration we have just to get it into the classification without doing a license review.. Sure, some part could be automated but not all.

For the users it would only add benefit if a lot of integrations are covered otherwise it's some they will learn to ignore.

fabaff commented 6 years ago

To be honest I'm a bit skeptical about the badges or the message they could send. If something is official then it doesn't mean that it's better than something unofficial or that the community thing is worse than the SDK of the vendor.

OverloadUT commented 6 years ago

Those badges would only be useful to developers to understand what is missing, which I don't think provides a ton of value. The simple Silver, Gold, Platinum provides immediate value to the end-user, and provides a good signal to developers about which components they use have room for improvement. Great way for new developers to find something to work on to learn the ropes too.

For me, I plan on creating GHIs like "ISY994 Road to Silver" which checkboxes for each requirement. Then I'll have a "Road to Gold" GHI when that one is done.

balloob commented 6 years ago

@OverloadUT is right, this scale is aimed at the end-user. The more levels we add, the less they will understand. Silver/Gold/Platinum is very straight forward.

I think that a "Maintained by manufacturer" badge would be a good idea. It will allow companies to distinguish themselves and help improve our system in the process.

That way we will have three things for each integrations:

fabaff commented 6 years ago

I guess that we have the first task for Hacktoberfest: Add Quality Scale :wink:

IoT class is already in place and "Maintained by manufacturer" is a very short list.

balloob commented 6 years ago

Yeah it will be fairly easy to spot gold/platinum. It's limited to https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/config_entries.py#L138