Temporary overrides for gates and configs

[matthewtole]

One of the features we need out of our gating / dynamic config system is for admins to be able to temporarily set an override for a given gate or config value. This is useful when testing out features during development, QA, and automated tests.

A simple solution to this could be providing an API for interacting with the gates themselves which could be used to add user IDs to a gate to override it to a particular value.

However it would be great if StatSig could provide an official mechanism for handling overrides (either in the SDK or at the API level).

Would love to hear the teams thoughts on this and what you recommend we use for this.

[marcos-statsig]

Hey Matthew, thanks for the feedback! For right now, would some gate configuration like in this screenshot work for your scenario?

You can add an ID condition to essentially override the gate for anyone given the user ID, and if you specifically want to have this override only apply on staging or test environments, you can add the Environment condition as well as shown in the screenshot.

If this approach doesn't work, or isn't optimal can you let us know more about your specific scenario to see how we can optimize this flow?

[matthewtole]

Yeah, that's the manual way of doing it, but at previous companies, we've had it built into dev & admin tools in the product to easily override gates without having to go into the UI. We can use the manual flow in the short term for some things, but this wouldn't work very well for automated testing flows.

[vijaye-statsig]

Matthew would these overrides be in the same session (meaning, override->check->unoverride)? Or would it be across multiple sessions (meaning override->run a bunch of tests for a while->unoverride)?

We aren't prioritizing mutation APIs but if it's something very specific we could solve it with a pointed solution.

[matthewtole]

Both would definitely be valuable in different situations, but having it be a single session would be fine.

The most immediate use case we have is the ability to just set the gate value for the current user directly from our web app. We have that ability right now with our current rudimentary gating system, and it's very useful to just be able to click a button in the admin menu and turn features on and off. Having to go into a 3rd party website, find the right gate, add a new override, and then remember or look up your user ID is a lot more friction than we currently have.

[jkw-statsig]

Hi @matthewtole - the scenario you described makes sense! Here is our proposal, let us know if you have any feedback:

• 3 new APIs, for our client SDKs only:
  1. overrideGate(gateName: string, value: boolean); no-op if the gateName does not exist (we'll display a console warn).
  2. clearOverrides(name: string?); clear all overrides if no name is provided, otherwise only clear the specified override
  3. getOverrides(); provides a list of pairs of existing overrides for the current user.
• The overrides are saved in local storage and persist across sessions for the user.

[matthewtole]

@jkw-statsig That sounds great! Just to clarify, from the perspective of the existing API, the overrides would be indistinguishable from the real values, and the getOverrides function is just to help with populating whatever UI we want to control the overrides?

[jkw-statsig]

Yes, checkGate will prioritize returning the value from overrides if it exists for the current user, and getOverrides makes it possible for you to show a list of overrides so the user can be aware what's being overridden and decides which one to remove.

Btw the name for api #2 should probably be removeOverride instead.

[matthewtole]

👍🏻

[jkw-statsig]

We've just released js SDK v3.0.0 that introduced these new APIs. See https://github.com/statsig-io/js-client/releases/tag/v3.0.0 for more details. Hope these will be helpful for you!

[tore-statsig]

@matthewtole this has been added to the react SDK in v0.4.0

You will have to use the useStatsig hook, but then all the same functionality of the statsig-js sdk will work

[jhurwitz]

Thanks for building this so quickly!

Just to confirm, it looks from the code like the overrides are stored locally; is that correct?

What's the override story in server SDKs? For unit tests it's easy enough to mock, but (as stated above) other use cases include QA and potentially integration tests where mocking is less straightforward.

I don't think we necessarily need server-side overrides in the near future, but it does seem a bit inconsistent to me that your initial suggestion was set overrides server-side on the StatSig website, but then the code change to make that flow easier only applies to client SDKs.

[jkw-statsig]

Hi @jhurwitz - yes the overriding methods for client SDKs are stored locally, which were done so so that the use case @matthewtole mentioned above where you'd want built-in menu options to override certain values with just a click can be done, as opposed to going to Statsig's console and change conditions there.

Our initial suggestions were more for automated testing, but it sounds like the being able to override locally for client SDK was more important based on @matthewtole's clarification after that.

Are what we offer now enough for your other use cases on the server-side, or would some additional server-side override features be helpful? We think the "environment tier" setting on the console is pretty useful for test environment overrides (you can override in one place and it works on all your tests), but if you can share some specific scenario/requirements where this doesn't support well, we'd be happy to provide better ways :).