BUG: frames to binary conversion is enforced when ignore_flyback>0 was previously set

Describe the issue:

I have used suite2p for multiplane imaging data and set ignore_flyback=2. In the first run, I get plane0 and plane1 processed, but plane2 ignored, just as expected. Running suite2p again without registration for the purpose of redoing roi detection, however, always leads to anew conversion from frames to binary. I digged a bit in the code and found that all 'suite2p/planeX' folders have to contain the data.bin file for suite2p to skip the conversion. 'plane2' does not contain any data.bin though, since ignored.

I believe the problem is in the run_s2p.py file.

elif len(plane_folders) > 0:
        ops_paths = [os.path.join(f, "ops.npy") for f in plane_folders]
        ops_found_flag = all([os.path.isfile(ops_path) for ops_path in ops_paths])
        binaries_found_flag = all([
            os.path.isfile(os.path.join(f, "data_raw.bin")) or
            os.path.isfile(os.path.join(f, "data.bin")) for f in plane_folders
        ])
        files_found_flag = ops_found_flag and binaries_found_flag

Reproduce the code example:

I run suite2p from the terminal with:

python run_suite2p.py {inputs} {outputs} --settings two_step_registration=True keep_movie_raw=True do_registration=2 nplanes=3 threshold_scaling=1.5 spatial_scale=2 do_registration=0

This is the script:

import matplotlib.pyplot as plt
import suite2p
import argparse
from pathlib import Path

if __name__ == "__main__":

    class ParseKwargs(argparse.Action):
        def __call__(self, parser, namespace, values, option_string=None):
            setattr(namespace, self.dest, dict())
            for value in values:
                key, value = value.split('=')
                getattr(namespace, self.dest)[key] = value

    parser = argparse.ArgumentParser(description="Run analysis script with datalad run")
    parser.add_argument('inputs', nargs='*', help='-i path/to/2p/dataset/subject/*drug/* (-i ...) -o path/to/suite2p/dataset/subject/combined_exp')
    parser.add_argument('outputs')
    parser.add_argument('--settings', nargs='*', action=ParseKwargs)
    args = parser.parse_args()

    data_path = args.inputs
    save_path0 = args.outputs

    # set the suite2p settings
    ops = suite2p.default_ops()

    # --- main ---
    # ops['nplanes'] = args.nplanes
    ops['tau'] = 1.25  # timescale of gcamp to use for deconvolution, 0.2 for GCaMP8m from the paper
    # ops['fs'] = 10  # sampling rate of recording, determines binning for cell detection, in Hz

    # --- output ---
    ops['reg_tif'] = True  # save registered tifs for manual segmentation and fissa
    ops['move_bin'] = False # make sure to ignor binary files and reg_tif to save time for datalad run cmd

    # --- registration ---
    ops['nimg_init'] = 600  # default 300, maybe higher helps with low signal-to-noise?
    ops['batch_size'] = 900  # we will decrease the batch_size in case low RAM on computer

    # --- ROI detection ---
    ops['roidetect'] = True
    ops['spatial_scale'] = 0  # 0=automatic, 1 (=6 pixels), 2 (=12 pixels), 3 (=24 pixels), or 4 (=48 pixels)
    ops['connected'] = False  # do not require ROIs to be fully connected as suggested for dendrites/boutons
    ops['threshold_scaling'] = 1.8  # we are increasing the threshold for finding ROIs to limit the number of non-cell
    # ROIs found (sometimes useful in gcamp injections)
    ops['max_overlap'] = 1.0  # to throw out no ROIs
    ops['max_iterations'] = 100
    ops['sparse_mode'] = True  # default
    ops['denoise'] = 0  # if True, sparse_mode has to be True, too

    # --- cellpose detection ---
    ops['anatomical_only'] = 0
    ops['diameter'] = 0

    # --- neuropil ---
    ops['allow_overlap'] = False

    # integrate any further custom settings
    if args.settings is not None:
        for key, val in args.settings.items():
            try:
                ops[key] = eval(val)
            except NameError:
                ops[key] = val

    if ops['nplanes'] == 3:
        ops['ignore_flyback'] = [2]
        ops['fs'] = 10

    data_path.sort()

    sub_exp = '/'.join(output_path.split('/')[-2:])
    fast_disk = f'/home/meng/Fast_disk/{sub_exp}'
    Path(fast_disk).mkdir(parents=True, exist_ok=True)
    db = {'data_path': data_path, 'save_path0': save_path0, 'fast_disk': fast_disk}
    output_ops = suite2p.run_s2p(ops=ops, db=db)

Error message:

...
** Found 468 tifs - converting to binary **
3600 frames of binary, time 16.63 sec.
7200 frames of binary, time 33.88 sec.
10800 frames of binary, time 51.08 sec.
14400 frames of binary, time 69.57 sec.
18000 frames of binary, time 97.42 sec.
21600 frames of binary, time 125.81 sec.
25200 frames of binary, time 153.15 sec.
28800 frames of binary, time 180.92 sec.
32400 frames of binary, time 208.34 sec.
36000 frames of binary, time 234.75 sec.
39600 frames of binary, time 260.59 sec.
43200 frames of binary, time 287.77 sec.
46800 frames of binary, time 314.89 sec.
50400 frames of binary, time 343.38 sec.
54000 frames of binary, time 370.69 sec.
57600 frames of binary, time 398.88 sec.
61200 frames of binary, time 426.36 sec.
64800 frames of binary, time 456.49 sec.
68400 frames of binary, time 484.39 sec.
72000 frames of binary, time 512.72 sec.
75600 frames of binary, time 540.16 sec.
79200 frames of binary, time 568.12 sec.
82800 frames of binary, time 596.18 sec.
86400 frames of binary, time 623.65 sec.
90000 frames of binary, time 651.55 sec.
93600 frames of binary, time 679.71 sec.
97200 frames of binary, time 707.78 sec.
100800 frames of binary, time 726.51 sec.
104400 frames of binary, time 743.08 sec.
108000 frames of binary, time 760.31 sec.
111600 frames of binary, time 777.21 sec.
115200 frames of binary, time 794.06 sec.
118800 frames of binary, time 811.21 sec.
122400 frames of binary, time 827.99 sec.
126000 frames of binary, time 844.62 sec.
129600 frames of binary, time 861.72 sec.
133200 frames of binary, time 878.14 sec.
136800 frames of binary, time 895.02 sec.
140400 frames of binary, time 912.36 sec.
time 912.81 sec. Wrote 46800 frames per binary for 3 planes
>>>>>>>>>>>>>>>>>>>>> PLANE 0 <<<<<<<<<<<<<<<<<<<<<<
NOTE: not running registration, ops['do_registration']=0
binary path: 01_data/02_preprocessed_data/02_Psychedelics_L5b_suite2p/231113D_Sim-Cre/Saline_LSD/suite2p/plane0/data.bin
NOTE: applying default /home/meng/.suite2p/classifiers/classifier_user.npy
----------- ROI DETECTION
Binning movie in chunks of length 12
...

Version information:

suite2p v0.14.3

Context for the issue:

No response

MouseLand / suite2p