extended_stimulus_presentations reward_rate calculation is incorrect

[alexpiet]

• rewarded doesn't match rewards table. I get this by comparing len(session.rewards) and np.sum(esp['rewarded'])
• units of reward_rate are in rewards/image, not rewards/second

[alexpiet]

@matchings @dougollerenshaw @yavorska-iryna This bug scares me because it calls into question other fields in extended stimulus processing! Yikes!

[matchings]

@alexpiet can you specify how you loaded the extended_stimulus_presentationstable to see this error? did you use loading.get_ophys_dataset(), then dataset.extended_stimulus_presentations? A code block to reproduce would be helpful

[dougollerenshaw]

I agree with @matchings. Some more specifics would be helpful.

[alexpiet]

Here is a minimal working example of the discrepancy for the "rewarded" column. Spot checking, it might only happen on some sessions.

oeid = 1010556662
session = loading.get_ophys_dataset(oeid, include_invalid_rois=False)
esp = session.extended_stimulus_presentations
print(np.sum(esp['rewarded']) # 164
print(len(session.rewards)) #138 

As far as the units for reward_rate, I determined that by visual inspection of the code, it needs to be divided by 0.75s like the lick_rate. You can verify this by loading the model output for a session, and plotting that reward -rate column

[dougollerenshaw]

OK, I'm able to replicate the mismatch in the reward count (164 vs 138). Investigating now. I had to add imports and fix one parentheses typo to get your code block to run. Here's a full working version for reference:

from visual_behavior.data_access import loading
import numpy as np

oeid = 1010556662
session = loading.get_ophys_dataset(oeid, include_invalid_rois=False)
esp = session.extended_stimulus_presentations

print(np.sum(esp['rewarded'])) # 164
print(len(session.rewards)) #138 

[alexpiet]

yes, sorry I dropped the imports. My head is in a different codebase right now, and I was just flagging this issue for later

[alexpiet]

How come your code block has nice colors, and mine doesnt?

[dougollerenshaw]

I just learned a new trick from @jsiegle recently. You can make github render your code with Python specific syntax highlighting by doing this:

Which gives:

import some_package

print(some_variable)

[dougollerenshaw]

Hmm, this is weird. Looking at extended stimulus presentations, I see some repeats. Look at rows 3/4 (index 60)