Beron2022 stickiness

[mobeets]

Fitting weights to predict the model's action $at \in \{-1, 1 \}$ as a function of $r{t-k}$, $a{t-k}$, and $y{t-k}$, for $k = 1, 2, ..., 5$.

• Model 1: To match what's in the paper, let $y_t = a_t r_t$. So $y_t = 1$ when ($a_t = 1$ and $r_t = 1$), and $y_t = -1$ when ($a_t = -1$ and $r_t = 1$). This is the orange model below.
• Model 2. I instead define $y_t = a_t \oplus r_t$, where $\oplus$ is such that $y_t = 1$ when ($a_t = 1$ and $r_t = 1$) OR ($a_t = 0$ and $r_t = 0$), and $y_t = 0$ when ($a_t = 0$ and $r_t = 1$) OR ($a_t = 1$ and $r_t = 0$). This is the blue model below.

Model 2 to me is more like the belief model. (Model 1 doesn't capture the trials where you learn about the unchosen arm.)

The interesting thing is that the action-history weights change sign when you use Model 2 vs. Model 1.

(Note: the above behavior came from using ε=0.1. But changing epsilon basically just changes the noise floor—it doesn't change the weight structure.)

[mobeets]

I realize now the logic for Model 1: Their $y_t$ covariate is only nonzero for rewarded actions—i.e., when the correct action was taken. I remember them saying something in the paper about "choice perseverance when the reward contingencies change," which is how you would interpret positive choice weights in Model 1.

The question is, what if you defined $y_t$ as in Model 2? Would you still see choice perseverance? I'd need to check the actual mouse data to answer this. But at least in the RNN, it's clear that Model 1 vs. Model 2 can influence the choice weights.

[mobeets]

I will post the plots later, but in the mouse data as well, the choice weights get higher if you use my definition of yt.

For the paper's definition, negative choice weights are the only way for the model to encode switches following an unrewarded choice. So there's a pressure for that first weight to be negative. So that's why the choice weights are more negative (ie lower) with their definition of yt than with mine.

From this perspective it seems like both the mice and rnn show perseveration. Though perhaps the mice show it more so than the rnn since you can see positive choice weights even with their definition.

[mobeets]

Mouse

[mobeets]

Mouse vs. RNN

Note: The new covariates, "A*(R-1)", encode whether a reward omission predicts a switch. This lets the choice history capture the perseveration bias and not also an omission-driven switch.

Both the mouse and the RNN appear to have a perseveration bias, but also a weaker omission-driven-switch.

[mobeets]

So after looking for this across multiple RNNs, most of them do show some "stickiness," but it's relatively small, and spread across all history weights.

An RNN fully trained on p=1, by contrast, shows zero stickiness. My suspicion is that the reward encoding matters. For example, if we indicated "r=0" with its own input, we might see different RNN solutions. Because theoretically, r=0 and r=1 are both equal amounts of information. But practically, the absence of an input might be harder for the RNN to interpret symmetrically. For example, to treat (A=1,R=0) the same as (A=0,R=1), as the belief model does.

So it might be interesting to train RNNs where R=0 is indicated by its own input, and compare those representations/choice decoding models. Maybe they are less sticky, and also have more symmetric responses.