Open kg3354 opened 1 month ago
Hello, Sorry for the late reply; I was outside the country for a conference.
There are several ways you can do this; I believe you probably can just pass in the list of original files instead of merging them, or if the filenames don't actually represent different sessions, you can do a flatter merge that doesn't preserve the distinctiveness of different filenames.
The former is probably better if the data came from multiple sessions (in which case you'll want to look at the demos where we cover how multisession registration works).
Cheers, Pat
Hi Pat, Thanks for reaching out! I solved that issue, now I am facing another one.
I am using tensorflow 2.4.1, CaImAn version 1.9.13, and python 3.8.0, and I am trying to use GPU during the CaImAn motion correction. I made sure to import pycuda.autoinit and used
motion_opts = opts.get_group('motion')
motion_opts['use_cuda'] = True
print(motion_opts)
logging.info(motion_opts)
mc = MotionCorrect(fnames, dview=dview, **motion_opts)
the MotionCorrect method in CaImAn returns an cuMemAlloc failed: initialization error. The detailed logs are:
6104 [motion_correction.py:motion_correction_piecewise():3218][112128] ** Starting parallel motion correction **
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/research/anaconda3/envs/fiola/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/research/anaconda3/envs/fiola/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/research/research/Corelink_FIOLA/k8s/init/motion_correction.py", line 3124, in tile_and_correct_wrapper
mc[count], total_shift, start_step, xy_grid = tile_and_correct(img, template, strides, overlaps, max_shifts,
File "/home/research/research/Corelink_FIOLA/k8s/init/motion_correction.py", line 2199, in tile_and_correct
rigid_shts, sfr_freq, diffphase = register_translation(
File "/home/research/research/Corelink_FIOLA/k8s/init/motion_correction.py", line 1700, in register_translation
image_gpu = gpuarray.to_gpu(np.stack((src_image, target_image)).astype(np.complex128))
File "/home/research/anaconda3/envs/fiola/lib/python3.8/site-packages/pycuda/gpuarray.py", line 1294, in to_gpu
result = GPUArray(ary.shape, ary.dtype, allocator, strides=_compact_strides(ary))
File "/home/research/anaconda3/envs/fiola/lib/python3.8/site-packages/pycuda/gpuarray.py", line 268, in __init__
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
pycuda._driver.LogicError: cuMemAlloc failed: initialization error
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "simple.py", line 223, in <module>
main()
File "simple.py", line 55, in main
caiman_file = run_caiman_init(fnames_init, pw_rigid=True, max_shifts=ms, gnb=nb, rf=15, K=4, gSig=[3, 3])
File "simple.py", line 103, in run_caiman_init
mc.motion_correct(save_movie=True)
File "/home/research/research/Corelink_FIOLA/k8s/init/motion_correction.py", line 273, in motion_correct
self.motion_correct_pwrigid(template=template, save_movie=save_movie)
File "/home/research/research/Corelink_FIOLA/k8s/init/motion_correction.py", line 372, in motion_correct_pwrigid
self.motion_correct_rigid()
File "/home/research/research/Corelink_FIOLA/k8s/init/motion_correction.py", line 317, in motion_correct_rigid
_fname_tot_rig, _total_template_rig, _templates_rig, _shifts_rig = motion_correct_batch_rigid(
File "/home/research/research/Corelink_FIOLA/k8s/init/motion_correction.py", line 2904, in motion_correct_batch_rigid
fname_tot_rig, res_rig = motion_correction_piecewise(fname, splits, strides=None, overlaps=None,
File "/home/research/research/Corelink_FIOLA/k8s/init/motion_correction.py", line 3220, in motion_correction_piecewise
res = dview.map(tile_and_correct_wrapper,pars)
File "/home/research/anaconda3/envs/fiola/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/research/anaconda3/envs/fiola/lib/python3.8/multiprocessing/pool.py", line 768, in get
raise self._value
pycuda._driver.LogicError: cuMemAlloc failed: initialization error
What might caused this error? Is there additional package I need to include?
Thanks for your time!
Kaiwen Guo
What kind of GPU are you running it on?
I am running it on Nvidia GeForce RTX 3070.
Thanks
Kaiwen Guo
On Mon, Jul 8, 2024 at 6:16 PM Pat Gunn @.***> wrote:
What kind of GPU are you running it on?
— Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_flatironinstitute_CaImAn_issues_1364-23issuecomment-2D2215443545&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=PBqA4ZL9u_nR0seDcrIZyQ&m=jD7MW6QVBCylObwL7uJ-OLlTQ9GuCxvdd4erY6xUanbNq85KM7gOny688_LFHRhV&s=BB01DFZ4YPiWU1dyTNCVN0rZWKOQvf3vLNOHs7M0rK0&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BDWJVAVEXVVLTBXZSVAFM43ZLMFUVAVCNFSM6AAAAABJK54YJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJVGQ2DGNJUGU&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=PBqA4ZL9u_nR0seDcrIZyQ&m=jD7MW6QVBCylObwL7uJ-OLlTQ9GuCxvdd4erY6xUanbNq85KM7gOny688_LFHRhV&s=X9SADFXHLsDFp-Jr4UYt-IWwY67K1PmN4J7HUFxPrTs&e= . You are receiving this because you authored the thread.Message ID: @.***>
Actually, I just spotted something above that surprised me; where and why did you import pycuda.autoinit?
The pycuda.autoinit is imported in the main script:
import caiman as cm
import logging
import matplotlib.pyplot as plt
import numpy as np
import os
import skcuda.cublas
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
os.environ["TF_XLA_FLAGS"] = "--tf_xla_enable_xla_devices"
import pyximport
pyximport.install()
import scipy
from tensorflow.python.client import device_lib
from time import time
import pickle
from fiola.fiolaparams import fiolaparams
from fiola.fiola import FIOLA
from fiola.utilities import download_demo, load, to_2D, movie_iterator
import tensorflow as tf
tf.debugging.set_log_device_placement(True)
physical_devices = tf.config.list_physical_devices('GPU')
if physical_devices:
tf.config.experimental.set_memory_growth(physical_devices[0], True)
else:
print("No GPU device found")
logging.basicConfig(format="%(relativeCreated)12d [%(filename)s:%(funcName)20s():%(lineno)s][%(process)d] %(message)s",
level=logging.INFO)
logging.info(device_lib.list_local_devices()) # if GPU is not detected, try to reinstall tensorflow with pip install tensorflow==2.5.0
startTime = time()
import cv2
import glob
from time import time
from caiman.motion_correction import MotionCorrect
from caiman.source_extraction.cnmf import cnmf as cnmf
from caiman.source_extraction.cnmf import params as params
from caiman.utils.utils import download_demo
from caiman.summary_images import local_correlations_movie_offline
from caiman.source_extraction.cnmf.utilities import get_file_size
import pycuda.autoinit
Without the pycuda.autoinit, i get the cuInit failed: initialization error as such:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 1675, in register_translation
cudactx # type: ignore
NameError: name 'cudactx' is not defined
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/research/anaconda3/envs/caimangpu/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/research/anaconda3/envs/caimangpu/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 3116, in tile_and_correct_wrapper
mc[count], total_shift, start_step, xy_grid = tile_and_correct(img, template, strides, overlaps, max_shifts,
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 2188, in tile_and_correct
rigid_shts, sfr_freq, diffphase = register_translation(
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 1677, in register_translation
init_cuda_process()
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 1389, in init_cuda_process
cudadrv.init()
pycuda._driver.LogicError: cuInit failed: initialization error
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "generate_init_result.py", line 324, in <module>
main()
File "generate_init_result.py", line 271, in main
caiman_file = run_caiman_init(fnames_init, pw_rigid=True, max_shifts=ms, gnb=nb, rf=15, K=4, gSig=[3, 3])
File "generate_init_result.py", line 92, in run_caiman_init
mc.motion_correct(save_movie=True)
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 259, in motion_correct
self.motion_correct_pwrigid(template=template, save_movie=save_movie)
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 358, in motion_correct_pwrigid
self.motion_correct_rigid()
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 303, in motion_correct_rigid
_fname_tot_rig, _total_template_rig, _templates_rig, _shifts_rig = motion_correct_batch_rigid(
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 2893, in motion_correct_batch_rigid
fname_tot_rig, res_rig = motion_correction_piecewise(fname, splits, strides=None, overlaps=None,
File "/home/research/research/CaImAn/caiman/motion_correction.py", line 3212, in motion_correction_piecewise
res = dview.map(tile_and_correct_wrapper,pars)
File "/home/research/anaconda3/envs/caimangpu/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/research/anaconda3/envs/caimangpu/lib/python3.8/multiprocessing/pool.py", line 768, in get
raise self._value
pycuda._driver.LogicError: cuInit failed: initialization error
Just for reference, the entire piece of code is:
#!/usr/bin/env python
import caiman as cm
import logging
import matplotlib.pyplot as plt
import numpy as np
import os
import skcuda.cublas
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
os.environ["TF_XLA_FLAGS"] = "--tf_xla_enable_xla_devices"
import pyximport
pyximport.install()
import scipy
from tensorflow.python.client import device_lib
from time import time
import pickle
from fiola.fiolaparams import fiolaparams
from fiola.fiola import FIOLA
from fiola.utilities import download_demo, load, to_2D, movie_iterator
import tensorflow as tf
tf.debugging.set_log_device_placement(True)
physical_devices = tf.config.list_physical_devices('GPU')
if physical_devices:
tf.config.experimental.set_memory_growth(physical_devices[0], True)
else:
print("No GPU device found")
logging.basicConfig(format="%(relativeCreated)12d [%(filename)s:%(funcName)20s():%(lineno)s][%(process)d] %(message)s",
level=logging.INFO)
logging.info(device_lib.list_local_devices()) # if GPU is not detected, try to reinstall tensorflow with pip install tensorflow==2.5.0
startTime = time()
import cv2
import glob
from time import time
from caiman.motion_correction import MotionCorrect
from caiman.source_extraction.cnmf import cnmf as cnmf
from caiman.source_extraction.cnmf import params as params
from caiman.utils.utils import download_demo
from caiman.summary_images import local_correlations_movie_offline
from caiman.source_extraction.cnmf.utilities import get_file_size
#import pycuda.autoinit
#%%
def run_caiman_init(fnames, pw_rigid=True, max_shifts=[6, 6], gnb=2, rf=15, K=5, gSig=[4, 4]):
c, dview, n_processes = cm.cluster.setup_cluster(
backend='local', n_processes=4, single_thread=False) # Adjust n_processes as needed
timing = {}
timing['start'] = time()
# dataset dependent parameters
display_images = False
fr = 30 # imaging rate in frames per second
decay_time = 0.4 # length of a typical transient in seconds
dxy = (2., 2.) # spatial resolution in x and y in (um per pixel)
patch_motion_um = (100., 100.) # patch size for non-rigid correction in um
strides = tuple([int(a/b) for a, b in zip(patch_motion_um, dxy)])
overlaps = (24, 24)
max_deviation_rigid = 3
mc_dict = {
'fnames': fnames,
'fr': fr,
'decay_time': decay_time,
'dxy': dxy,
'pw_rigid': pw_rigid,
'max_shifts': max_shifts,
'strides': strides,
'overlaps': overlaps,
'max_deviation_rigid': max_deviation_rigid,
'border_nan': 'copy',
}
opts = params.CNMFParams(params_dict=mc_dict)
# Motion correction and memory mapping
time_init = time()
motion_opts = opts.get_group('motion')
motion_opts['use_cuda'] = True
print(motion_opts)
logging.info(motion_opts)
mc = MotionCorrect(fnames, dview=dview, **motion_opts)
mc.motion_correct(save_movie=True)
logging.info('Finished mc, before cm.save_memmap')
border_to_0 = 0 if mc.border_nan == 'copy' else mc.border_to_0
fname_new = cm.save_memmap(mc.mmap_file, base_name='memmap_', order='C', border_to_0=border_to_0) # exclude borders
logging.info('Finished exclude borders')
Yr, dims, T = cm.load_memmap(fname_new)
images = np.reshape(Yr.T, [T] + list(dims), order='F')
logging.info('Finished reshaping borders')
# Restart cluster to clean up memory
cm.stop_server(dview=dview)
c, dview, n_processes = cm.cluster.setup_cluster(backend='local', n_processes=4, single_thread=False) # Adjust n_processes as needed
logging.info('Finished restarting cluster')
f_F_mmap = mc.mmap_file[0]
Cns = local_correlations_movie_offline(f_F_mmap, remove_baseline=True, window=1000, stride=1000, winSize_baseline=100, quantil_min_baseline=10, dview=dview)
logging.info('Finished local_correlations_movie_offline')
Cn = Cns.max(axis=0)
Cn[np.isnan(Cn)] = 0
plt.imshow(Cn, vmax=0.5)
# Parameters for source extraction and deconvolution
p = 1
merge_thr = 0.85
stride_cnmf = 6
method_init = 'greedy_roi'
ssub = 2
tsub = 2
opts_dict = {'fnames': fnames,
'p': p,
'fr': fr,
'nb': gnb,
'rf': rf,
'K': K,
'gSig': gSig,
'stride': stride_cnmf,
'method_init': method_init,
'rolling_sum': True,
'merge_thr': merge_thr,
'n_processes': n_processes,
'only_init': True,
'ssub': ssub,
'tsub': tsub}
opts.change_params(params_dict=opts_dict)
cnm = cnmf.CNMF(n_processes, params=opts, dview=dview)
cnm = cnm.fit(images)
logging.info('Finished CNFM on patches')
# Component evaluation
min_SNR = 1.0
rval_thr = 0.75
cnn_thr = 0.3
cnn_lowest = 0.0
cnm.params.set('quality', {'decay_time': decay_time,
'min_SNR': min_SNR,
'rval_thr': rval_thr,
'use_cnn': False,
'min_cnn_thr': cnn_thr,
'cnn_lowest': cnn_lowest})
cnm.estimates.evaluate_components(images, cnm.params, dview=dview)
print(len(cnm.estimates.idx_components))
time_patch = time()
if display_images:
cnm.estimates.plot_contours(img=Cn, idx=cnm.estimates.idx_components)
cnm.estimates.select_components(use_object=True)
cnm2 = cnm.refit(images, dview=dview)
time_end = time()
print('Total time until cnm refit is: ')
print(time_end - time_init)
min_SNR = 2
rval_thr = 0.85
cnn_thr = 0.15
cnn_lowest = 0.0
cnm2.params.set('quality', {'decay_time': decay_time,
'min_SNR': min_SNR,
'rval_thr': rval_thr,
'use_cnn': False,
'min_cnn_thr': cnn_thr,
'cnn_lowest': cnn_lowest})
cnm2.estimates.evaluate_components(images, cnm2.params, dview=dview)
print(len(cnm2.estimates.idx_components))
cnm2.estimates.plot_contours(img=Cn, idx=cnm2.estimates.idx_components)
if display_images:
cnm2.estimates.view_components(images, img=Cn, idx=cnm2.estimates.idx_components)
cnm2.estimates.view_components(images, img=Cn, idx=cnm2.estimates.idx_components_bad)
cnm2.estimates.select_components(use_object=True)
cnm2.estimates.detrend_df_f(quantileMin=8, frames_window=250)
cnm2.mmap_F = f_F_mmap
cnm2.estimates.Cn = Cn
cnm2.estimates.template = mc.total_template_rig
cnm2.estimates.shifts = mc.shifts_rig
save_name = cnm2.mmap_file[:-5] + '_caiman_init.hdf5'
timing['end'] = time()
print(timing)
cnm2.save(save_name)
print(save_name)
output_file = save_name
cm.stop_server(dview=dview)
log_files = glob.glob('*_LOG_*')
for log_file in log_files:
os.remove(log_file)
plt.close('all')
return output_file
def main():
mode = 'calcium'
if mode == 'calcium':
folder = os.getenv('MOVIE_FOLDER', '/usr/src/app')
fnames = folder + '/test_sub.tif'
fr = 30
num_frames_init = 2000
num_frames_total = 30000
offline_batch = 100
batch = 100
flip = False
detrend = False
dc_param = 0.9995
do_deconvolve = True
ms = [3, 3]
center_dims = None
hals_movie = 'hp_thresh'
n_split = 1
nb = 1
trace_with_neg = True
lag = 5
options = {
'fnames': fnames,
'fr': fr,
'mode': mode,
'num_frames_init': num_frames_init,
'num_frames_total': num_frames_total,
'offline_batch': offline_batch,
'batch': batch,
'flip': flip,
'detrend': detrend,
'dc_param': dc_param,
'do_deconvolve': do_deconvolve,
'ms': ms,
'hals_movie': hals_movie,
'center_dims': center_dims,
'n_split': n_split,
'nb': nb,
'trace_with_neg': trace_with_neg,
'lag': lag
}
mov = cm.load(fnames, subindices=range(num_frames_init))
fnames_init = fnames.split('.')[0] + '_init.tif'
mov.save(fnames_init)
print(tf.test.gpu_device_name())
caiman_file = run_caiman_init(fnames_init, pw_rigid=True, max_shifts=ms, gnb=nb, rf=15, K=4, gSig=[3, 3])
if __name__ == "__main__":
main()
Thank you for your time
Kaiwen Guo
I'm not that familiar with fiola; I've heard of it but never used it.
Can you start debugging by trying to run the notebooks included with caiman? Knowing if those work will help us figure out where the issue starts.
Hi Pat,
I tried debugging using the caiman notebooks, and different errors rises.
The commands I followed are
mamba create -n gpu5 -c conda-forge caiman
conda activate gpu5
caimanmanager install
I was able to pass all the tests in cpu mode. Then i used
conda update opencv
conda install -c conda-forge pycuda
and set use_cuda to true in demo_pipeline.py. The new error message I get is
ModuleNotFoundError: No module named 'skcuda'
My fix then is to get the newest skcuda package from their github repo, and I used
git clone https://github.com/lebedov/scikit-cuda.git
cd scikit-cuda
pip install .
And when i run the demo_pipeline.py again, it says:
pycuda._driver.LogicError: cuFuncSetBlockShape failed: invalid resource handle
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/research/caiman_data/demos/general/demo_pipeline.py", line 202, in <module>
main()
File "/home/research/caiman_data/demos/general/demo_pipeline.py", line 85, in main
mc.motion_correct(save_movie=True)
File "/home/research/miniconda3/envs/gpu5/lib/python3.11/site-packages/caiman/motion_correction.py", line 244, in motion_correct
self.motion_correct_pwrigid(template=template, save_movie=save_movie)
File "/home/research/miniconda3/envs/gpu5/lib/python3.11/site-packages/caiman/motion_correction.py", line 343, in motion_correct_pwrigid
self.motion_correct_rigid()
File "/home/research/miniconda3/envs/gpu5/lib/python3.11/site-packages/caiman/motion_correction.py", line 288, in motion_correct_rigid
_fname_tot_rig, _total_template_rig, _templates_rig, _shifts_rig = motion_correct_batch_rigid(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/research/miniconda3/envs/gpu5/lib/python3.11/site-packages/caiman/motion_correction.py", line 2844, in motion_correct_batch_rigid
fname_tot_rig, res_rig = motion_correction_piecewise(fname, splits, strides=None, overlaps=None,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/research/miniconda3/envs/gpu5/lib/python3.11/site-packages/caiman/motion_correction.py", line 3149, in motion_correction_piecewise
res = dview.map(tile_and_correct_wrapper,pars)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/research/miniconda3/envs/gpu5/lib/python3.11/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/research/miniconda3/envs/gpu5/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
I also tried to install CUDA first, then use pip to install pycuda, but it gives me the same error.
Hello, Caiman itself can use the GPU without needing additional software installs; at most you might need to sidegrade the tensorflow versions using conde.
But I see that fiola uses pycuda.
One thing to try is to use conda versions of all these packages when possible - try very hard to avoid using pip, and to use gpu/cuda-enabled versions of all the same packages. That will prevent version incompatibilities.
pycuda._driver.LogicError: cuFuncSetBlockShape failed: invalid resource handle
My guess is this is depcreated.
I'm a bit confused, what is your usecase? scikit-cuda
is very old and not maintained AFAIK, last release was ~10 years IIRC. I would be surprised tensorflow v2 and scikit-cuda.
If you want to use FIOLA for online CNMF(E) it does not use scikit-cuda either.
Hi Kushal,
I am currently only trying to use GPU on CaImAn's motion correction part of demo_pipeline.py. The scikit-cuda
pops up when I installed pucuda via conda and set use_cuda
to true. The FIOLA part works well for now, its just that MC on CPU takes a lot of time and CaImAn seems to support GPU for that section.
Im also at NYU btw :)
Thanks, Kaiwen Guo
If you just want gpu motion correction I'd strongly recommend jnormcorre, it's under active development: https://github.com/apasarkar/jnormcorre
The gpu accelerated motion correction in caiman is very old.
Hi,
I am trying to use caiman to get a initialization file so that I can reuse it in the future. I am passing in a tif file generated from a sequence of avi file, and the error message that i get is that
I wonder if it is due to the way I construct the tif file. I made sure it is not colored, and here is the script i used to generate the tif file:
My script that calls caiman is:
and the demo_initialize_calcium.py corresponding to run_caiman_init:
Sorry for throwing out chunks of code. I am new to caiman, and any help would be really apprecaited!
Your setup: