Error in loop cnmfe

[zhouyi0812]

Hello!!

I was trying to looping cnmfe by integrating the code together. However, sometimes it works, but most of the time it finished process the movie but the batch was not successful.

Here is my code and wonder if you could give me some guidance on how to adjust it? Thank you!

import os
os.environ["MESMERIZE_N_PROCESSES"] = "15"
print(os.environ["MESMERIZE_N_PROCESSES"])

import os
from pathlib import Path 
import numpy as np
import matplotlib.pyplot as plt

mc.set_parent_raw_data_path("C:/Users/yyizhou/caiman_data/")
parent_raw_data_path = mc.get_parent_raw_data_path()

# Directory containing the movie files
movie_dir = Path(parent_raw_data_path).joinpath("Movie/d7")

# Specify the name of the movie to start from (without the .tif extension)
start_from_movie = "016C5d7_B1_F2_1_00d00h00m"
# Flag to indicate whether processing should start
start_processing = False

gSig = 5

# Parameters for the algorithm
params_cnmfe = {
    "main": {
        'method_init': 'corr_pnr',  # use this for 1 photon
        'K': None,
        'gSig': (gSig, gSig),
        'gSiz': (4 * gSig + 1, 4 * gSig + 1),
        'merge_thr': 0.75,
        'p': 1,
        'tsub': 1,
        'ssub': 1,
        'rf': 50,
        'stride': 25,
        'only_init': True,  # set it to True to run CNMF-E
        'nb': 0,
        'nb_patch': 0,
        'method_deconvolution': 'oasis',  # could use 'cvxpy' alternatively
        'low_rank_background': None,
        'update_background_components': True,  # sometimes setting to False improve the results
        'normalize_init': False,  # just leave as is
        'center_psf': True,  # leave as is for 1 photon
        'ssub_B': 1,
        'ring_size_factor': 1.4,
        'min_corr': 0.75,  # min peak value from correlation image
        'min_pnr': 6.5,
        'del_duplicates': True,  # whether to remove duplicates from initialization
        'bas_nonneg': True,
    }
}

# Loop over each .tif movie file in the directory
for movie_file in movie_dir.glob('*.tif'):
    movie_name = movie_file.stem  # Extract the movie name without the extension

    # Check if this is the movie to start from or if we're already processing
    if movie_name == start_from_movie or start_processing:
        start_processing = True  # Start processing if the current movie is the specified one or afterwards
    else:
        continue  # Skip to the next movie if we haven't reached the starting point yet

    try:
        # Construct the batch path dynamically based on the movie name
        batch_path = parent_raw_data_path.joinpath(f"mesmerize-cnmfe/{movie_name}/batch.pickle")

        # Load the batch corresponding to the current movie
        df = mc.load_batch(batch_path)

        print(f"Start processing cnmfe: {movie_name}")
        mcorr_movie = df.iloc[0].mcorr.get_output()

        # Assuming LazyTiff and correlation_pnr are defined and available in your context
        a = LazyTiff(df.iloc[0].caiman.get_input_movie_path())
        def load_weird_movie(path):
            return LazyTiff(path, shape=(540, 1152, 1536))

        corr, pnr = correlation_pnr(mcorr_movie[::1], gSig=gSig, swap_dim=False)

        df.caiman.add_item(
            algo="cnmfe",
            input_movie_path=movie_file,  # Directly use movie_file as the path
            params=params_cnmfe,
            item_name=movie_name  # Assuming this is how you intend to name the item
        )

        for i, row in df.iterrows():
            if row["outputs"] is not None:  # item has already been run
                continue  # skip

            process = row.caiman.run()

            # "DummyProcess" is used for local backend so this is automatic
            if process.__class__.__name__ == "DummyProcess":
                df = df.caiman.reload_from_disk()
            print(f"Finished processing cnmfe: {movie_name}")

    except Exception as e:
        print(f"Error processing {movie_name}: {e}. Skipping to next movie.")
        continue

And the notification of running the cnmfe is: Start processing cnmfe: 016C5d7_B1_F2_1_00d00h00m Running 25ed74f2-722e-480d-8cd3-934e9e1a36fa with local backend cnmfe params: {'main': {'method_init': 'corr_pnr', 'K': None, 'gSig': (5, 5), 'gSiz': (21, 21), 'merge_thr': 0.75, 'p': 1, 'tsub': 1, 'ssub': 1, 'rf': 50, 'stride': 25, 'only_init': True, 'nb': 0, 'nb_patch': 0, 'method_deconvolution': 'oasis', 'low_rank_background': None, 'update_background_components': True, 'normalize_init': False, 'center_psf': True, 'ssub_B': 1, 'ring_size_factor': 1.4, 'min_corr': 0.75, 'min_pnr': 6.5, 'del_duplicates': True, 'bas_nonneg': True}} The local backend is an alias for the multiprocessing backend, and the alias may be removed in some future version of Caiman making memmap Performing CNMFE Finished processing cnmfe: 016C5d7_B1_F2_1_00d00h00m

Thank you very much!!

[kushalkolar]

If a batch item doesn't work it's usually due to the parameters.

Also, you don't need to create a new batch dataframe for each movie. I'd recommend learning pandas, you can filter based on values in columns etc.

[zhouyi0812]

Hello! May I ask if there is a way that I can get the optimal cor/pnr from the motion correction movie out from my 1500 movie efficiently? Like if there is a code that could detect the peak from the plot? Thank you very much!

[kushalkolar]

You don't want the peak, you want the values such that the intersection of corr and pnr images captures neurons and excludes background/other activity. It's hard to get around visualization and be sure that you're not just capturing non-neuronal components.