Create shim for decrosstalk QC data

When discussing #149 with Natalia, she expressed concern that the code she uses to prototype visualizations and other QC operations on the decrosstalking output expects QC data in the old schema and that she did not have time to update it to reflect the new schema. I believe it is straightforward to create a class that presents the same data access API as the MesocopeICA class from her mesoscope_ica repository, but that takes its input from the new HDF5 file produced by the decrosstalking pipleline (once #207 gets merged). I had hoped to make this work a part of #207, but the baseline QC output and visualization implementation took more time than I expected. This ticket exists to capture the work of creating that shim between the new QC output and Natalia's existing API, if we decide that is still valuable to do.

A little more detail on what this work would entail.

Natalia's prototyping repository is here

https://github.com/AllenInstitute/mesoscope_ica

Her visualization code exists in mesoscope_ica/plotting.py within that repository. The plotting script includes a lot of methods that proceed like

from mesoscope_ica.unmixing import MesoscopeICA as ICA

....

def plot_all_rois_in_session(session, cache):
    """
    function to plot roi figures for all cell ids in current session
    :param session: LIMS session ID
    :param cache: directory here outputs are read/written
    :return: None
    """

    ica = ICA(session_id=session, cache=cache, roi_name='roi', np_name='neuropil')
    pairs = ica.dataset.get_paired_planes()  # array of pairs in this dataset

    for pair in pairs:

        update_ica(ica, pair)
        tkey = 'roi'

        for pkey in ica.pkeys:  # for two planes in a pair:

            exp_id = ica.exp_ids[pkey]
            pkey_coupled = [it for it in ica.pkeys if it != pkey]
            exp_id_coupled = ica.exp_ids[pkey_coupled[0]]
            # check that experiment exists in experiments_table

            # get traces
            traces_out_active = {}
            traces_in = ica.raws[pkey][tkey]
            traces_out = ica.outs[pkey][tkey]
            traces_in_active = ica.raws_active[pkey]
            traces_in_active = ica.get_ica_active_events(traces_in, traces_in_active)  # 2 x active trace length
            ....

The MesoscopeICA class is defined in mesoscope_ica/unmixing.py it both runs the decrosstalking algorithm and provides access to the resulting QC data products.

The scope of this ticket would be to create a class that mirrors all of the getter methods in MesoscopeICA but that, instead of getting the data by actually performing the calculation (or reading data based on the old QC schema), gets the data by reading the new QC schema, which is defined in the README.md in ophys_etl_pipelines/src/modules/decrosstalk/

AllenInstitute / ophys_etl_pipelines

Create shim for decrosstalk QC data #208