search

assaf / vanity

Experiment Driven Development for Ruby
http://vanity.labnotes.org
MIT License
1.55k stars 269 forks source link

use_vanity not working? #360

Closed daniel-nelson closed 5 years ago

daniel-nelson commented 5 years ago

After upgrading Vanity to 3.0.0 (with Rails 5.1.6.2), I noticed that use_vanity wasn't working.

Previously: use_vanity :current_vanity_user

where current_vanity_user returned

OpenStruct.new(id: current_user.vanity_id)
# also tried current_vanity_user returning the following
OpenStruct.new(vanity_identity: current_user.vanity_id)

With that, even though a user had been opted into a particular variant (as follows), Vanity.ab_test(:my_experiment) was returning 'invisible' in a Rails view: Vanity.context = OpenStruct.new(vanity_identity: user.vanity_id) Vanity.playground.experiment(:my_experiment).chooses('visible')

Switching to a custom method in a before_action fixed it (confirmed that when I switch the user between the 'visible' and 'invisible' variants, Vanity.ab_test(:my_experiment) returns the correct variant in a Rails view): before_action :set_vanity_context

def set_vanity_context
  Vanity.context = current_vanity_user
end

# where current_vanity_user returns
OpenStruct.new(vanity_identity: current_user.vanity_id)

We haven't run any Vanity tests for a while, and it's possible it never worked for us and we just never noticed. But since I'm noticing it now, I figured I'd report it. We can get around it easily by just using the before_action.

This does bring up some confusion about whether we should be returning something that responds to id or vanity_identity. https://github.com/assaf/vanity/blob/master/doc/identity.textile seems to indicate it's looking for id; whereas, the readme states Vanity pulls the identity from a "context" object that responds to vanity_identity.

phillbaker commented 5 years ago

Sorry to hear that things aren't working as intended! It's been a busy few days and I won't have much time to look at this this week.

Trom your perspective, could this be better documented or do you think we should change some code to handle this?

daniel-nelson commented 5 years ago

We went back to using use_vanity because we needed to be able to opt into experiments using _vanity. Then, opting in was working when using the vanity id from the session, so I don't know what was going on. I do know that opting in from the command line wasn't having any effect unless I made the change indicated above.

Anyway, our marketing person wants us to switch to Google Optimizer so that we can more closely control who is counted towards any given experiment, so it looks like we won't be using Vanity to run experiments, at least for now.

Thanks for all your work on this! We've loved it through the years.