issue with data processing for MESO2 sessions

[matchings]

I am attempting to do some analysis with the initial datasets from the new Mesoscope (MESO.2) and i am encountering some weird issues where some of the experiments (imaging planes) in a given session have a normal number of 2P frames (around 48,000), while others have exactly 10,000. This makes me think that the data is getting truncated to some default value at some point in the processing where it has failed. Perhaps during demixing, since it only applies to some planes in a given session?

Here is a table with the number of 2p frames (determined based on length of dff traces) for each session / experiment that i have looked at from MESO2: "\allen\programs\braintv\workgroups\nc-ophys\visual_behavior\meso2_2_frame_issue.csv"

And here are the values directly:

@nataliaorlova is also interested in the solution to this issue as she is managing the PLC process for MESO2 and needs to know the outcome of these experiments

[wbwakeman]

Pika to investigate. The goal of the investigation will be to understand the issue and have an idea about how to solve it. If the solution requires a code change, that should be documented in this or a new issue. The effort should be timeboxed to 2 days or less.

Tasks:

• [ ] Determine why the 10000 frame files are being created, and how it may be related to the non-converging experiments
• [ ] Explain to Sam/Natalia/Marina/Pika what is happening
• [ ] Work with them to understand the desired behavior for the various cases that might be encountered.

INFO: Using cached SIF image
ERROR:ophys_etl.modules.decrosstalk.decrosstalk:ICA did not converge for any ROIs when applying decrosstalk to ophys_experiment_id: 1090898031 (1090898029)
ERROR:ophys_etl.modules.decrosstalk.decrosstalk:ICA did not converge for any ROIs when applying decrosstalk to ophys_experiment_id: 1090898034 (1090898032)
ERROR:ophys_etl.modules.decrosstalk.decrosstalk:ICA did not converge for any ROIs when applying decrosstalk to ophys_experiment_id: 1090898032 (1090898034)
ERROR:ophys_etl.modules.decrosstalk.decrosstalk:ICA did not converge for any ROIs when applying decrosstalk to ophys_experiment_id: 1090898037 (1090898035)
ERROR:ophys_etl.modules.decrosstalk.decrosstalk:ICA did not converge for any ROIs when applying decrosstalk to ophys_experiment_id: 1090898035 (1090898037)

[danielsf]

I suspect this is the block of code producing those garbage traces with 10,000 time steps

https://github.com/AllenInstitute/ophys_etl_pipelines/blob/main/src/ophys_etl/modules/decrosstalk/__main__.py#L125-L139

If I understand my own code correctly, these are dummy traces written out when literally no ROIs converged during decrosstalk, which is a known research problem (i.e. scientists/data scientists need to decide what to do when decrosstalking fails to converge)

[matchings]

@nataliaorlova is there something different about the data from meso2 that would cause so many failures to converge? Maybe the SNR in those planes is especially low?

It seems to me that the key here is having a way to know whether data going in to decrosstalk is of acceptable quality or not (other than by just looking at it).

If we are in a low SNR situation, failure to converge shouldn’t be surprising. But if the data is good and it’s failing to converge, we may need to optimize the algorithm somehow (if that’s even possible).

[wbwakeman]

@nataliaorlova @matchings Just checking in on this. Let us know how you want to move forward to address this based on information provided in above comments.