search

markhwhiteii / bwsTools

Tools for Case 1 Best-Worst Scaling (MaxDiff) Designs
https://osf.io/wb4c3/
Other
13 stars 2 forks source link

hb-mnl? #10

Open markhwhiteii opened 4 years ago

markhwhiteii commented 4 years ago

I can't get it to run quickly enough: https://discourse.mc-stan.org/t/troubleshooting-a-maxdiff-hierarchical-multinomial-regression/11235

andytimm commented 1 year ago

I am comfortable speeding up Stan code, but learning about MaxDiff, so was thinking contributing here can be a reasonable way to learn.

Could you confirm you’d still review a PR here if I make it in the next few months? Since it’s been a while since this package been worked on wanted to ask early.

markhwhiteii commented 1 year ago

Yes, definitely would review a PR here. Haven't worked on it recently because I'm gonna have to rework the unit tests that are on codecov, and I've been busy with work/other projects/life. But would definitely love the help, and I'm willing to chat about anything MaxDiff generally.

andytimm commented 1 year ago

Ok, well figured out what was wrong with the model as coded this week.

The main issue was that to_vector(Theta) ~ normal(0, 10); with as many choices as in the example is implausibly wide. Translating that to the logit scale implies a pretty strong belief that 1 brand is likely to get 90%+ of the choice. You can see some good graphs of this phenomenon in Jim Savage's slides here on 18/19: https://rpubs.com/jimsavage/sawtoothcon. Another win for the folk theorem.

Fitting with that knocked down to N(0,.5) takes only about 45 minutes (one chain was much slower than the others, most were done in ~25 - not sure if that is because I was also running other stuff and that core was being hit harder). The resulting model seems pretty sensible, and the values of Theta align with fitting EB to the same data pretty well (EG healthcare is the best message).

I'll do a bit more digging and see if I can't improve on this more, but it does seem like this was the main problem. Since the appropriate sigma in the prior depends on the number of choices, I'll probably try and implement something like the priors here next: https://journals.sagepub.com/doi/abs/10.1509/jmkr.46.6.832_JMR6J?journalCode=mrja&utm_source=pocket_saves.

I'm thinking I'll probably do an initial PR that just implements the model without covariates first, then add in covariates after you confirm there's nothing in the implementation you want modified to fit the package design ideas. Sound good?

markhwhiteii commented 1 year ago

Yeah, that all sounds incredible! Good finds. Definitely would be ideal to have some sort of code that takes number of choices and chooses a sensible default prior. Thanks.