BUG: ops['subfolders'] ignored when loading in GUI

[samupicard]

I am having issues specifying subfolders in the ops file. I'm generating an ops.json in matlab and loading this in suite2p. In this json,ops['look one level down']=True and the ops['subfolders'] are specified as a list of strings (I have attached an example). Yet, when loading this ops file in the suite2p GUI, somehow this list gets overwritten by the default empty list contained in db, as revealed by the print(db) command below. As a result it complains that no tiffs were found.

I have made a temporary workaround in run_s2p.py (line 340-346) but i would appreciate your help in fixing this issue more permanently. Many thanks!!

ops.zip

t0 = time.time()
subfolders = ops['subfolders'] #SP quick patch
ops = {**default_ops(), **ops, **db}
if isinstance(ops['diameter'], list) and len(ops['diameter'])>1 and ops['aspect']==1.0:
    ops['aspect'] = ops['diameter'][0] / ops['diameter'][1]
print(db)
ops['subfolders']=subfolders #SP quick patch

Error message:

{'data_path': ['M:/Subjects/SP035/2023-02-24/001'], 'subfolders': [], 'h5py_key': 'data', 'save_path0': 'M:/Subjects/SP035/2023-02-24/alf', 'fast_disk': 'C:/Users/Samuel/Documents/2PI/Subjects/SP035/2023-02-24', 'input_format': 'tif'}
NOTE: nplanes 1 nrois 6 => ops['nplanes'] = 6
mesoscan
['M:/Subjects/SP035/2023-02-24/001']
Could not find any tiffs TEST
>>>ERROR<<<
Traceback (most recent call last):
  File "C:\Users\Samuel\anaconda3\envs\suite2p\lib\runpy.py", line 194, in _run_module_as_main
>>>ERROR<<<
    return _run_code(code, main_globals, None,
  File "C:\Users\Samuel\anaconda3\envs\suite2p\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\users\samuel\suite2p\suite2p\__main__.py", line 78, in <module>
>>>ERROR<<<
    main()
  File "c:\users\samuel\suite2p\suite2p\__main__.py", line 71, in main
    run_s2p(ops, db)
  File "c:\users\samuel\suite2p\suite2p\run_s2p.py", line 404, in run_s2p
>>>ERROR<<<
    ops0 = convert_funs[ops['input_format']](ops.copy())
  File "c:\users\samuel\suite2p\suite2p\io\tiff.py", line 281, in mesoscan_to_binary
    ops1, fs, reg_file, reg_file_chan2 = utils.find_files_open_binaries(ops1, False)
  File "c:\users\samuel\suite2p\suite2p\io\utils.py", line 224, in find_files_open_binaries
    fs, ops2 = get_tif_list(ops1[0])
  File "c:\users\samuel\suite2p\suite2p\io\utils.py", line 161, in get_tif_list
    raise Exception('no tiffs')
Exception: no tiffs
Interrupted by error (not finished)

Version information:

suite2p v0.12.1.dev14+g2579413

[samupicard]

After some digging it turns out that any keys in db overwrites keys set in ops, but in the gui i don't think we can access this db?

[chriski777]

Hi @samupicard, I believe it should be possible to specify a db.npy in the same folder as your ops.json file. Can you try creating a db.npy in the same directory with the db[subfolders] set to your subfolders and try running this again?

[samupicard]

Does the db file have to be a numpy array? I am asking because I'm currently scripting everything in matlab so it would be really helpful not to have to run a separate python script just for this purpose.

[samupicard]

Are there other keys this db.npy file should contain (other than db['subfolders'])? Even when I specify subfolders, data_path, save_path0, fast_disk and save to the same directory as ops.json, the process run from the GUI is still defaulting to 'subfolders': [] ...

checking the content of my db.npy file:

In [17]: import numpy as np
In [18]: np.load('db.npy',allow_pickle=True)
Out[18]: array({'data_path': ['M:/Subjects/SP035/2023-02-24/001'], 'subfolders': ['raw_imaging_data_01', 'raw_imaging_data_03'], 'save_path0': 'M:/Subjects/SP035/2023-02-24/001/alf', 'fast_disk': 'C:/Users/Samuel/Documents/2PI/Subjects/SP035/2023-02-24/001'}, dtype=object)

running suite2p from GUI loading the ops.json file (with the db.npy in the same folder):

{'data_path': ['M:/Subjects/SP035/2023-02-24/001'], 'subfolders': [], 'h5py_key': 'data', 'save_path0': 'M:/Subjects/SP035/2023-02-24/001/alf', 'fast_disk': 'C:/Users/Samuel/Documents/2PI/Subjects/SP035/2023-02-24/001', 'input_format': 'tif'}
NOTE: nplanes 1 nrois 6 => ops['nplanes'] = 6
mesoscan
['M:/Subjects/SP035/2023-02-24/001']
Could not find any tiffs
>>>ERROR<<<
Traceback (most recent call last):
  File "C:\Users\Samuel\anaconda3\envs\suite2p\lib\runpy.py", line 194, in _run_module_as_main
>>>ERROR<<<
    return _run_code(code, main_globals, None,
  File "C:\Users\Samuel\anaconda3\envs\suite2p\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\users\samuel\suite2p\suite2p\__main__.py", line 78, in <module>
    main()
  File "c:\users\samuel\suite2p\suite2p\__main__.py", line 71, in main
    run_s2p(ops, db)
  File "c:\users\samuel\suite2p\suite2p\run_s2p.py", line 406, in run_s2p
    ops0 = convert_funs[ops['input_format']](ops.copy())
  File "c:\users\samuel\suite2p\suite2p\io\tiff.py", line 281, in mesoscan_to_binary
>>>ERROR<<<
    ops1, fs, reg_file, reg_file_chan2 = utils.find_files_open_binaries(ops1, False)
  File "c:\users\samuel\suite2p\suite2p\io\utils.py", line 223, in find_files_open_binaries
>>>ERROR<<<
    fs, ops2 = get_tif_list(ops1[0])
  File "c:\users\samuel\suite2p\suite2p\io\utils.py", line 160, in get_tif_list
>>>ERROR<<<
    raise Exception('no tiffs')
Exception: no tiffs
Interrupted by error (not finished)

[chriski777]

Hi @samupicard, sorry for the late response. For the time being, I would suggest running this on a notebook to address this need. There is indeed a bug in the GUI that causes the db parameter you're providing to be overwritten... Since this involves the GUI, this will take some time to address. Sorry about that!