Model Recovery

[mdsteiner]

Program model recovery.

[mdsteiner]

Here are some plots from the simulations (simulated data, no fitting is done yet):

For the Equal Environment

Likelihood of a risky choice by Trial

Likelihood of a Risky Choice by Points

For the Low Environment

Likelihood of Risky Choice by Trial

Likelihood for a Risky Choice by Points

And for the High Environment

Likelihood of a Risky Choice by Trial

Likelihood of a Risky Choice by Points

Note

For each environment 400 subjects (100 for each model) were simulated. The data is in the ModelSimDat data files.

[ndphillips]

cool! But I'm a bit confused...if these are simulated data, then why do we observe non--random choices for the Random model. I'm specifically looking at the final graph (Likelihood of a risky choice by points). Is this an issue with the specific model underlying geom_smooth()? Are you using the loewess method? Maybe try another method that doesn't assume any underlying model (like the running mean). There must be a method like this programmed into geom_smooth

[mdsteiner]

I think it may be an artefact because when we look at the plots with trial on the x axis the random model really chooses random. But I think that only if you pick the risky options and are very lucky you'll reach > 120 or so points, so the forms of these curves might make sense. There are also much fewer observations in these point ranges.

Yes I've used loess, but I'll see if there's also another method

[ndphillips]

ahhhh, yes that makes sense! Given that a random agent has earned many points, it must have (by chance) selected the high variance option a lot and gotten lucky. Similar with random agents that earn very few points.

man that's a really fun example of a non-intuitive finding. I'll have to remember that :)

[mdsteiner]

Here are the new plots for the simulations with the new model included.

For the Low Environment

For the High Environment

For the Equal Environment

[ndphillips]

Thanks. Quick question about what is being plotted on the y-axis. Is this...

A: the probability of selecting a risky option on trial T given that an agent has X points on trial T - 1?

Or is it

B: the proportion of all trials where agents select risky options given that they earned X points across all trials?

If it is (A) then we should definitely not see any pattern with the random agents. That is, the probability of selecting a risky option on trial T should be completely independent of how many points the agent had on trial T - 1

If it is (B), then we could see such a pattern, because agents that, by chance, selected risky options a lot, might have more (or less) points than agents who, by chance, did not.

If you're in the office today, we can also chat :)

[ndphillips]

Also, I think it would be good to restrict the x-axis on all plots to [-20, 120]. It should be very rare for people to have point values outside of this range (this is the case with the experimental data right?) and as there's such little agent data in this range, we shouldn't try to interpret it much

[mdsteiner]

Not in the office, I have to organise things for moving...

Actually what I did was to create 10 bins of points along the range of min(points.cum) to max(points.cum) of a person and took the mean of risky choices of that bin. I then took the mean of points.cum of a bin to have a value for the x axis. This is how the finer lines were created. The mean lines are then done with geom_smooth() with loess.

Yes I agree abouth the range of points, I'll definitely change that if we include such a plot somewhere.

[mdsteiner]

So the model recovery finally ran through. Here are the results

Percent Correct Classifications:

	Corr Classified
NaturalMean	0.937
Random	0.702
RL	0.994
RLGoal	1.000
SampEx_Int_Goal	1.000

Parameter Recoveries (RLGoal Curvature and Loss Aversion Parameters are not shown)

Looks pretty reasonable to me...

[ndphillips]

yes there are great results indeed! Definitely include these in the updated results section :)

One quick question: Does percent correct classifcations mean p(Correct classification | True model) or p(Correct classification | Fitted Model)? I am guessing it is conditional on the true model...either way, make sure to clarify this.

[mdsteiner]

it means p(Correct classification | True model). I will include a description of what we find and include the plot and table in the supplement (I think we already have too many plots and tables, although I had the plan to have one panel of plots with the pirateplot and the pRisky and mean points per trial at the top row and the three environments of pRisky by Points in the bottom row, but I need to find a way of how to get base plots and ggplot plots in the same panel)