search

Betterment / test_track

Server app for the TestTrack multi-platform split-testing and feature-gating system
MIT License
119 stars 33 forks source link

Augment split finish migration to capture final decision #74

Open jmileham opened 6 years ago

jmileham commented 6 years ago

We hope to add support for mobile clients in the field who reference since-finished splits and provide a final disposition for those splits.

In order to provide the correct answer to those clients, we need to reliably capture that decision, so coupling it to the finish migration seems like the right choice.

This probably means adding a nullable decision column to splits. Note that this same column could probably be shared by an insta-decide feature for non-finished splits as well.

smudge commented 6 years ago

Could also work for web clients that still reference the split, right?

jmileham commented 6 years ago

i want to break web clients because it's the dev's fault then. :)

jmileham commented 6 years ago

(and you can force upgrade of the web client)

smudge commented 6 years ago

You'd remove the web client code before you decide the split? What if you need to quickly reverse the feature?

I was thinking about one case in particular where, right after flipping a feature on for everyone, the web client would essentially flip the feature off sporadically because of errors/timeouts in talking to TT. If the client had a way of knowing about final decisions, it could perhaps be a bit smarter in that scenario. We'd want to think it through of course (you could end up with behavior that lags behind whatever you're telling TestTrack to do). But that's where my head was at. Obv. after a while, if the client is still requesting a split, we should provide some form of strong encouragement for devs to remove the references in the code.

jmileham commented 6 years ago

Deciding and finishing are different, right, so we'd want to be able to decide immediately and then finish in a follow-up migration later. But I do like the idea of being able to pull the conditionals out in the same PR that has the finish migration in it, which this wouldn't allow unless we do ... something? So keep thinking about it and if you've got a great answer, lez do it.

I think operating TT reliably enough that you don't get timeouts on active splits is kind of key. I'm also not super interested in baking a persistent cache into the non-mobile clients?

jmileham commented 6 years ago

I guess after-release migrations would be the ticket. rails-core has talked about that a bit, I think. Maybe we should adopt a tool that does that and make that the best practice?

smudge commented 6 years ago

Yeah after-release or next-release migrations would be my answer there. It's the same kind of feature I'd want for DB schema changes.

jmileham commented 6 years ago

Yeah, I'd think we'd ride on activerecord after-release migrations

smudge commented 6 years ago

Alright, so I can buy that the story around what to do during in-flight rollouts is to not have your TestTrack go a little sideways, and that the small comfort of handling that scenario more gracefully is not worth the added complexity. But if it happens more than a little, despite our best efforts, we should revisit, because having a 'B' case unexpectedly flip back to 'A' can also be a risk (i.e. more than just an annoyance), and is not something I'd expect devs to account for.

jmileham commented 6 years ago

https://github.com/Betterment/test_track_rails_client/issues/56 should address that flapping issue! Thanks for reminding me to post that idea.