Open marcdotson opened 3 years ago
Updated comparison of results for simulated data with both ANA and respondent quality estimated with a heterogenous, 1000-member ensemble:
Model | LOO | Hit Rate | Hit Prob |
---|---|---|---|
HMNL | -2691 | 0.453 | 0.391 |
Ensemble (LOO Weights) | -2652 | 0.464 | 0.366 |
Ensemble (Equal Weights) | -2714 | 0.444 | 0.361 |
Ensemble (MNL Weights w/Predicted Probs) | -2715 | 0.444 | 0.361 |
Ensemble (Simple Count Weights) | -2711 | 0.444 | 0.361 |
Ensemble (Simple Probability Weights) | -2710 | 0.444 | 0.361 |
Updated comparison of results for real data where we account for both ANA and respondent quality with a heterogeneous, 1000-member ensemble:
Model | LOO | Hit Rate | Hit Prob |
---|---|---|---|
HMNL | -2756 | 0.403 | 0.348 |
Ensemble (LOO Weights) | -2771 | 0.417 | 0.296 |
Ensemble (Equal Weights) | -2870 | 0.414 | 0.291 |
Ensemble (MNL Weights w/Predicted Probs) | -2872 | 0.380 | 0.287 |
Ensemble (Simple Count Weights) | -2867 | 0.380 | 0.287 |
Ensemble (Simple Probability Weights) | -2834 | 0.387 | 0.289 |
And to check for any issues with how we are currently implementing the respondent quality pathology, here are updated results for simulated data with ANA only with a heterogenous, 1000-member ensemble:
Model | LOO | Hit Rate | Hit Prob |
---|---|---|---|
HMNL | -2787 | 0.411 | 0.368 |
Ensemble (LOO Weights) | -2723 | 0.410 | 0.347 |
Ensemble (Equal Weights) | -2789 | 0.414 | 0.349 |
Ensemble (MNL Weights w/Predicted Probs) | -2772 | 0.414 | 0.350 |
Ensemble (Simple Count Weights) | -2790 | 0.419 | 0.349 |
Ensemble (Simple Probability Weights) | -2785 | 0.414 | 0.349 |
And here are results for real data where we account for ANA only with a heterogeneous, 1000-member ensemble:
Model | LOO | Hit Rate | Hit Prob |
---|---|---|---|
HMNL | -2756 | 0.367 | 0.336 |
Ensemble (MNL Weights w/Predicted Probs) | -2923 | 0.380 | 0.286 |
Ensemble (Simple Count Weights) | -2925 | 0.380 | 0.286 |
Ensemble (Simple Probability Weights) | -2890 | 0.387 | 0.288 |
FWIW, @RogerOverNOut I've re-run the model with just ANA for both simulated and real data. You said it might be easier to work with that when trying your alternative meta-learner. I had to make some much clearer names. See the shared folder.
Thanks Marc
On Fri, Jun 4, 2021, 2:31 AM Marc Dotson @.***> wrote:
FWIW, @RogerOverNOut https://github.com/RogerOverNOut I've re-run the model with just ANA for both simulated and real data. You said it might be easier to work with that when trying your alternative meta-learner. I had to make some much clearer names. See the shared folder.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marcdotson/conjoint-ensembles/issues/68#issuecomment-854399843, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHEP55B2FNQKO7T5VHS2JL3TRBXKLANCNFSM45N7IGWA .
@jeff-dotson @RogerOverNOut I made my first pass using an MNL as a meta-learner. The results are in the above tables under "Ensemble (Logit Weights)."
I haven't done this before, so I wanted to describe what I'm doing. Please let me know if there's a red flag you see:
predictive_fit_stacking()
based on Roger's predictive_fit_ensemble()
to produce choice predictions for each of the ensemble members.mnl.stan
.MNL using probabilities instead of the choices has been added to the tables above. I've also added weights using simple counts of the hits as well as a sum of probabilities for the hits. Note that weights produced using simple counts of the hits is the same as weights from an MNL meta-leaner using probabilities.
Using {logitr} for the meta-learner would help boost speed.
Using LOO for model stacking produces improvement in terms of LOO only for the conjoint ensemble. What about alternate meta-learners? Use the
meta-learner
branch.