drbenvincent / darc_toolbox

Run adaptive decision making experiments
MIT License
16 stars 2 forks source link

Implement parameter recovery simulations #6

Closed drbenvincent closed 5 years ago

drbenvincent commented 5 years ago

In order to have faith that everything is working, we need to be able to perform some parameter recovery simulations.

The goal

Be able to reproduce figures like this

screen shot 2018-08-17 at 13 36 46

Basic functionality

Technical debt to clear

Parameter recovery

drbenvincent commented 5 years ago

current results: screen shot 2018-11-26 at 14 05 03 We are getting some discrepant, and unexpected behaviours that deviate from the Matlab results. Todo:

Check that the Python model code is actually using the fixed value of alpha=2 and not in fact inferring it's values.

drbenvincent commented 5 years ago

Actually, much of the discrepancy was because our prior on logk was waaaaay more broad than it should have been.

drbenvincent commented 5 years ago

This issue is now resolved. We can do parameter recovery and parameter recovery sweeps and get some good outputs. Further improvements to our BAD approach will be dealt with elsewhere :)

drbenvincent commented 5 years ago

For the record, the relatively poor performance of our approach was being caused by the heuristic (random_choice_dimension='DB'). Once removed, the estimates for our approach are back to being very good.