CNMF Fails on Windows with 'need at most 63 handles, got a sequence of length 65' Error

[Tepps-ps]

I am encountering an error while running CNMF on Mesmerize on my Windows PC equipped with 32 cores (Threadripper 5975). I managed to bypass a similar issue for motion correction by adjusting the n_processes variable to 50. However, CNMF fails even when I change the n_processes variable to 1. Below is the output for CNMF in Mesmerize, alongside the parameters I used:

`Running 55b8a326-8c06-4ef0-8018-e6c8cb791515 with local backend

---

Starting CNMF item: algo cnmf item_name nXCaMP08_002denoised-1 input_movie_path 25db67f8-2c25-44ea-a8f7-ebdb6964adce\25db67f8-2c25-44ea-a8f7-ebdb6964adce-nXCaMP08_002denoised-1_els__d1_256_d2256... params {'main': {'fr': 15, 'p': 1, 'nb': 2, 'merge_thr': 0.85, 'rf': 60, 'stride': 10, 'K': 4, 'gSig': (4, 4), 'ssub': 1, '... outputs None added_time 2024-03-11T14:06:52 ran_time None algo_duration None comments None uuid 55b8a326-8c06-4ef0-8018-e6c8cb791515 Name: 7, dtype: object With params:{'main': {'fr': 15, 'p': 1, 'nb': 2, 'merge_thr': 0.85, 'rf': 60, 'stride': 10, 'K': 4, 'gSig': (4, 4), 'ssub': 1, 'tsub': 1, 'method_init': 'greedy_roi', 'min_SNR': 7.5, 'rval_thr': 0.7, 'use_cnn': True, 'min_cnn_thr': 0.8, 'cnn_lowest': 0.1, 'decay_time': 4}, 'refit': False} The local backend is an alias for the multiprocessing backend, and the alias may be removed in some future version of Caiman making memmap WARNING:caiman.cluster:The local backend is an alias for the multiprocessing backend, and the alias may be removed in some future version of Caiman Exception in thread Thread-12 (_handle_workers): Traceback (most recent call last): File "C:\Users\jnlab\anaconda3\envs\mescore\lib\threading.py", line 1016, in _bootstrap_inner self.run() File "C:\Users\jnlab\anaconda3\envs\mescore\lib\site-packages\ipykernel\ipkernel.py", line 761, in run_closure _threading_Thread_run(self) File "C:\Users\jnlab\anaconda3\envs\mescore\lib\threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "C:\Users\jnlab\anaconda3\envs\mescore\lib\multiprocessing\pool.py", line 522, in _handle_workers cls._wait_for_updates(current_sentinels, change_notifier) File "C:\Users\jnlab\anaconda3\envs\mescore\lib\multiprocessing\pool.py", line 502, in _wait_for_updates wait(sentinels, timeout=timeout) File "C:\Users\jnlab\anaconda3\envs\mescore\lib\multiprocessing\connection.py", line 884, in wait ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout) File "C:\Users\jnlab\anaconda3\envs\mescore\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait res = _winapi.WaitForMultipleObjects(L, False, timeout) ValueError: need at most 63 handles, got a sequence of length 65 performing CNMF fitting images performing eval GPU run not requested, disabling use of GPUs USING MODEL (keras API): C:\Users\jnlab\caiman_data\model\cnn_model.json 1/1 [==============================] - 0s 208ms/step`

I've noticed the error mentions needing at most 63 handles but got a sequence of length 65. I'm curious if anyone has insights or suggestions on how to address this issue. Any help would be greatly appreciated!

[kushalkolar]

Did you set the environment variable? https://github.com/nel-lab/mesmerize-core/issues/205#issuecomment-1574585472

If that doesn't help then I'm not sure if CNMF is spawning more processes somewhere, but anyways this is really a problem with windows: https://github.com/scikit-learn/scikit-learn/issues/13354

[Tepps-ps]

CNMF failed even after changing the n_processes variable to 1. The simple solution is to split the file, but can I use multi-session registration in mesmerize?

[kushalkolar]

splitting the file isn't going to reduce the number of processes being used, you might want to try a non-windows machine or a computer with fewer cores

[Tepps-ps]

I see. I ran cnmf on the same file with the old mesmerize and it seemed to complete the process properly.

[kushalkolar]

that uses a very old version of caiman, I dunno I don't have the bandwidth to dig into this further.

[Tepps-ps]

Thx anyway.