How to deal with multiple environments, including local development?

[bremnes]

I notice that some other feature toggling services, such as LaunchDarkly, supports multiple environments.

What is the recommended way of dealing with feature flags through the code life cycle with Unleash? Perhaps it exists, but I couldn't find it in the documentation.

For instance:

• When developing locally, are you supposed to spin up your private unleash instance? I'd imagine using a shared instance would lead to a lot of "garbage" after a while. (Easily done by using the docker containers for unleash + pg)
• When having multiple environments - dev, qa, prod, feature X and so on:
  • Should they have separate unleash instances? I'd assume so.
  • Are feature flags supposed to be transferred between unleash instances manually? I get that the default value set in code would be used if the feature flag is not defined in Unleash. So either activate the flag in the environment yourself or give a heads-up to for instance the product owner that the feature is ready to be tested at their own discretion.
• Are you somehow turning off/on or "transferring" flags w/value between environments in Unleash through the CI/CD pipeline? I'd imagine this could be messy if you are working with feature branches and a lot of parallel environments. But it could be something "fancy", like adding comments to the pull request or through slack: /feature ShowMaintenanceMode true. (Perhaps not that easy for more complex activation strategies.)
• LaunchDarkly has a blog article regarding Strategies for Testing with Feature Flags . Care to share how you are handling testing with feature flags and Unleash?

I do realize that some of these questions are general for feature flags and not specific for Unleash. But it would by helpful with some examples, knowing that bigger organizations like Finn and Nav uses it.

[ivarconr]

tldr; We aim for short lived feature toggles, and use it as one of the tools which allows us to ship code to production in small batches at high speed. We aim to have new features enabled in dev and during integration testing as soon as we start exposing the feature to new users.

What is the recommended way of dealing with feature flags through the code life cycle with Unleash? Perhaps it exists, but I couldn't find it in the documentation.

I must admit that Unleash does not have sufficient support for multiple environments today. But there is some possibilities via the activation strategies. Can can create a custom activation strategy which will enable the feature toggle in defined environments, and rely on the other strategies to control roll-out in production.

However I think this can be better addressed by another way to think about activation strategies, where you also can set limiting conditions. This would allow users to have one setup of Unleash serving all environments. This is something I have on my schedule to address after I have completed #379.

But I guess it depends on your situation. For example in FINN we have just two environments dev and prod, and we have one instance of Unleash in each environment. We use feature toggles as one of the tools which allows us to ship new code all the time in small batches and thus avoids the need to have a diverging environments. Typically only the activation strategies are different in each environment.

Are feature flags supposed to be transferred between unleash instances manually?

When we introduce a new feature to production we typically want it disabled in production during deploy and enable it after the code has been successfully rolled out. The Unleash UI will even inform you about toggles used in an application that is not defined in the unleash-service.

In our situation the team and the product owner work closely together and creating the new toggle in production is an integral part of rolling out a new feature. Developers and product owners are eager to test their new features in production before enabling for real users.

When developing locally, are you supposed to spin up your private unleash instance? Care to share how you are handling testing with feature flags and Unleash?

For unit testing and local development one approach is to swap out the implementation, e.g. the Java SDK has a FakeUnleash implementation. This can be used in unit tests and during local development, and is typically managed via some sort of DI. Wiremock, nock is other viable approaches during testing.

Are you somehow turning off/on or "transferring" flags w/value between environments in Unleash through the CI/CD pipeline?

We don't do that. It should be possible to do with more advanced tools such as spinnaker using the Unleash API's (e.g. should be fairly easy to copy one toggle configuration in dev and apply it to prod) I'm not sure if this is a good idea however. You probably also have to think about roll-back, and my gut feeling is that you only ending up with a even more complex deploy pipeline.

@bremnes I would love t hear more about your thoughts on how to better address environments.

• Does it make sense to have a single setup of Unleash, where you can easier control activation based on environment?
• ...or do you think it would be preferable to have one Unleash per environment? How would like to move configuration between environments? What should be Unleash's role in this? Would Unleash then need to support rolling back changes? Create snapshots? Apply diffs?
• ...or maybe Unleash should have multiple namespaces and you are able to promote toggles between the namespaces?

[bremnes]

Thanks for the detailed reply, @ivarconr! (Sorry for the late feedback - vacation)

I wasn't aware that the UI displays feature toggles that are requested from the client, but aren't defined in the unleash service itself. Browsing into an application on my Unleash demo instance shows it just like in your post - neat! Then it should be a bit easier to navigate which (new) features that are toggable or not.

For your questions - I don't have "the answer" as I'm unsure about these things. From what I've read there are a lot of approaches, which led me to asking how you guys do it :)

Does it make sense to have a single setup of Unleash, where you can easier control activation based on environment?

I'm not sure about this one as I can see pros and cons on both approaches. With one shared instance it's less infrastructure to take care of, but if you have temporary environments and toggles it could be a bigger job to maintain/trim. In addition you have the availability/security aspect of all environments using one shared instance.

...or do you think it would be preferable to have one Unleash per environment? How would like to move configuration between environments? What should be Unleash's role in this? Would Unleash then need to support rolling back changes? Create snapshots? Apply diffs?

I'm not sure I'd want too much magic involved here. I see that LaunchDarkly have integration with multiple ci/cd, for instance Azure devops/VSTS, where you can associate a feature flag with a release. But I don't know how they handle rollback or re-release of an earlier version. But if we start to "bundle" feature flags too hard to releases, then I feel like we're on a slipery slope where we just as well could do it in regular code or configuration instead of an external toggling system such as Unleash. Not sure if that makes sense. https://launchdarkly.com/blog/microsoft-vsts-and-launchdarkly-team-up-to-control-releases/ https://docs.launchdarkly.com/docs/visual-studio-team-services-extension

...or maybe Unleash should have multiple namespaces and you are able to promote toggles between the namespaces?

I think I would prefer this way if environments were to be shared on one instance. This would give you some control and isolation, and all feature flags wouldn't be mixed together. So if I were to create a new feature-branch-environment I could clone the dev-environment and its feature toggles and play around (automatically when creating/tearing down the infrastructure for the designated code branch). But I'd imagine one would have to think about what to share between environments: ACL, activation strategies, feature toggles, metrics and so on.

[kjdelisle]

The way we've been doing it at STACK thus far is to give the application an environment variable for its "app name" (usually pulled from package.json, but can be overridden), and then pair that with the feature toggle in question. So, for the multiplexer service, we might have toggles with these names:

multiplexer-dev-abc123.use-new-encryption
multiplexer-staging.use-new-encryption
multiplexer-production.use-new-encryption

It does mean that we're doing the "namespace" management ourselves, but it hasn't involved much in the way of extra code on the app side of things, and it's worked well so far.

Note: We do run separate Unleash servers for our logical environments, but this strategy has let us do things like deploy temporary parallel environments to diagnose issues in-flight without having to worry about how to differentiate between toggles on those environments.

It would definitely be nice to see some built-in namespacing, though. :)

[ivarconr]

Thanks for sharing!

It would definitely be nice to see some built-in namespacing, though. :)

Agree, I have started the process on this. I believe it is possible without to much rework.

[deepakhj]

Gitlab has the idea of environments and a default case for feature flags. They use Unleash under the hood. Unfortunately it's a paid feature only. https://docs.gitlab.com/ee/user/project/operations/feature_flags.html

[ivarconr]

442 solves this.

[batjko]

442 solves this.

Isn't the MR still open, or why was this issue already closed?

[nemonemi]

FYI, the feature flag option is now free in Gitlab https://gitlab.sirius.supplyon.com/help/operations/feature_flags