Traces returned by BehaviorOphysExperiment.corrected_fluorescence_traces are not neuropil corrected

[aamster]

The documentation in the code states that these traces are neuropil corrected, however they are just demixed.

Talk with science team about this issue, and whether we need to actually return the neuropil corrected traces rather than just demixed.

[matchings]

@aamster I believe we should release both the demixed traces (what is currently provided as corrected_fluorescence_traces) and the neuropil corrected traces, as their own attribute. Ideally this would include the r-value used in the calculation of neuropil traces (as in https://github.com/AllenInstitute/AllenSDK/issues/1843) and the neuropil masks. These could all be in the same table, with r-value and masks as additional columns. Alternatively, the neuropil masks could go in the cell_specimen_table along with the main cell ROI masks.

If we had these additional artifacts, we could have more easily detected the problems with neuropil correction that @jkim0731 has detected and is working through in this issue: https://github.com/AllenInstitute/LearningmFISHTask1A_data_validation/issues/15 . For example, the r-value should never be greater than 1, but Jinho found many instances of this by digging through the relevant files in lims. A potential cause of this could be that there is a bright dendrite or other parts of a neuron in the neuropil masks, resulting in real signal being subtracted from the main ROI and throwing off the r-value (and subsequent dFF calculation). Being able to easily visualize the neuropil masks would allow us (and external users) to diagnose such issues. Right now there is very little insight into these processing steps via SDK.

@danielsf This should probably be added to the list of VBO re-release tasks. Apologies for not remembering it during the meeting yesterday.

[danielsf]

@matchings

It sounds like, from your comment above, you want us to add functionality to the SDK to return some new trace quality metrics. Can you open a new ticket specifying what they are. That work is probably independent of "make the SDK return demixed and neuropil-subtracted traces" separately.

[matchings]

@danielsf depends on what you mean by "new trace quality metric". The r-value metric that I am talking about is saved in the 'neuropil_correction.h5' in the experiment directory in lims, so it doens't need to be computed de novo. Its a value that is inherently associated with the neuropil corrected traces. I can make a new issue though.