Add beta feature toggle

[bafonins]

Summary

It would be great to introduce beta features toggle in webui.

Why do we need this?

We often add new features or perform big changes for existing features. This often causes issues when deploying to production, like buggy behavior, regressions to properly working functionality, incorrect handling when dealing with real data and other. Also, think of partial implementation of features that do not cover all usage cases.

Proper testing can address some of the points from above, however covering every possible user action with automated scripts is very tedious and in fact can introduce more friction later in development when extending existing functionality. Additionally, testing every user input is rather unrealistic when taking into account all browsers and devices that the users might use to access the Console.

I propose to tag certain features in the Console under the beta tag. Similar to how it is done in the v2 Console:

By default beta features wont be visible to the users, but it should be possible to enable this. I think adding a checkbox to the user settings, something like Enable beta features should be enough @kschiffer :

IMO, adding/extending features under the beta tag can drastically improve the whole development process of the Console, as well as clearly communicate to the users that beta features can be changed/removed/broken/incomplete.

Currently, we have the freedom to make any changes to webui, but later with increased number of users/customers, PCN this wont be the case anymore.

What is missing? What do you want to see?

Actual implementation from the How do you propose to implement this? section Documentation cc @laurensslats @benolayinka

Environment

3.7.2+

How do you propose to implement this?

As mentioned before, we should add a dedicated page with functionality to enable beta features. Note, there is no api for this and it is not required. We can simply store this option in the user attributes, something like {"console.ui.use-beta": "true"} should work just fine. Another options could be localStorate, but in this case users preference cannot be saved across multiple devices.

For the rest, we can introduce a react hook use-beta:

const [beta, setUseBeta] = useBeta();

And withBeta hoc for react class components.

Can you do this yourself and submit a Pull Request?

yes

cc @johanstokking

[johanstokking]

Yeah I think having a way to use and evaluate features while reducing the chance of stability regressions is a good aim.

We can discuss a few aspects of this;

• Will there be a "beta" and "stable" version of the same feature? If yes, that means we might need two or more versions of the same functionality in the code base
• How do we go with things that require an API change and the compatibility commitment? Would we have a "beta" or "experimental" mark also on the API, even if this is part of a new release?
• If it's specifically about the Console, wouldn't it be simpler if we had a way to request a specific Console version by query string or something, that informs the proxy to address a specific set of containers?
• How does it relate to having a separate staging environments that runs latest master or even develop builds?

[bafonins]

Will there be a "beta" and "stable" version of the same feature? If yes, that means we might need two or more versions of the same functionality in the code base

Yes. Basically, I decided to dicuss because of https://github.com/TheThingsNetwork/lorawan-stack/issues/579. Current device form works fine for most of the cases and PR for with the wizard component changes not only the UI, but the way we handle device (in the Console). Taking into account that device creation is one of the core functionality we should be careful here.

We can always lazy load beta features UI to avoid increased bundle size if this is a problem.

Also, I envision features not reside in the beta state for long. I would say 1 release, 2 max.

How do we go with things that require an API change and the compatibility commitment? Would we have a "beta" or "experimental" mark also on the API, even if this is part of a new release?

This issue is solely about the Console (or the whole webui package in general if required). In case of API changes, I dont think we should mark features that require this particular API as 'beta'.

If it's specifically about the Console, wouldn't it be simpler if we had a way to request a specific Console version by query string or something, that informs the proxy to address a specific set of containers?

Not sure, how does this help? I dont think that the users of PCN or CH will go through the changelog to find the exact version they need to, say, revert a particular feature. Correct me if I am missing something.

How does it relate to having a separate staging environments that runs latest master or even develop builds?

Staging environment helps a lot when real data is necessary to test/evaluate features. This issue is about more thorough process of landing new features to give the opportunity for early adopters/enthusiasts to get hands on the latests features while not affects other users.

[htdvisser]

Looking back at the regressions we had recently, I don't think those could have been avoided with feature toggles. For (partial/experimental) implementations of new features I do think feature toggles are useful.

However, these new/changed features are rarely limited to only the web UI, so I think it would be short-sighted to only implement the feature flags there. We need to make sure that both the backend and the frontend understand that a new/different implementation should be used for a specific user, gateway or application.

Some time ago I started looking into flagr for this. It looks pretty nice and has libraries for Go and JS.

See also https://github.com/TheThingsIndustries/lorawan-stack/issues/1639

---

If all we want to do is test new console features, it's a relatively small effort to add an additional console to a deployment under <domain>/betaconsole, and have that run builds from the master or develop branch.

[kschiffer]

I agree a beta toggle would give us some insight and reassurance when releasing big features, so that's indeed a good thing to have.

IMO, for now we can achieve similar benefits with a separate beta console, like @htdvisser suggested, while being a lot simpler to set up, enabling us to put our focus more on setting up a foundation of end-to-end tests. A more sophisticated beta toggle system would be a good aim for the PCN, where we would have a larger user-base and hence more beta-users to obtain meaningful feedback.

What we can also already do with minimal effort, is label new complex features with the beta tag by default, without any opt-in/out in the settings, like in the v2 console. This only works for new features and not updates though.

Regarding a fully featured beta toggle system, some challenges I see are:

• Determining whether a feature should be feature toggled or not
  • We should have a clear policy for that, not everything should start as beta by default
• Added complexity to releases
  • Possibly forgetting to un-beta features
  • The toggle mechanic itself leading to bugs
  • Increased complexity of end-to-end testing
• Having to keep two separate implementations in case of updated features
  • As noted, in some cases that might be impossible if the change relies on an API change
  • Lazy loading can mitigate file size here indeed
• Ensuring proper feedback from beta-users
  • We should establish a proper feedback loop, encouraging users to submit feedback when they use beta features and allowing them to do so easily
  • We could tag beta-users in Sentry to be able to filter on that

All of these should be solvable though.

[bafonins]

wont implement