search

kushalkolar / MESmerize

Platform for Calcium Imaging analysis. DEPRECATED.
GNU General Public License v3.0
59 stars 14 forks source link

[BUG] Error running full CNMFe #82

Closed WhereWolf72 closed 2 years ago

WhereWolf72 commented 2 years ago

Describe the bug I can import a .tif file using the tiff importer and json_minimal metadata importer. Further, I can inspect the correlation and PNR after running that preliminary analysis. However, when I go to run the full CNMFe analysis, I get an error and don't get an output .hdf5 file, but get the other output files that I have gotten previously upon successful CNMFe analysis. The only updates to my machine have been regarding the OS (Ubuntu 20.04), so I'm not sure exactly what has changed that has caused this issue as I have run this analysis before without problem.

To Reproduce

  1. open viewer
  2. modules > load images > tiff file
  3. select file and load asarray, select json file and use json_minimal metadata importer
  4. modules > CaImAn Toolbox > CNMF-E
  5. input gSig size, enter name, and click "add to batch"
  6. Go to batch manager, run the task, double click on task to inspect correlation and PNR values
  7. save correlation/PNR image as .png
  8. return to CNMF-E window, adjust min_corr and min_pnr values accordingly, enter name for analysis, and click "add to batch"
  9. go to batch manager, run the task
  10. Upon notification that task has been run, task will be highlighted in red and no .hdf5 file will be output, disallowing visualization of ROIs. Seemingly all other file outputs are present in working directory.

Expected behavior Upon running full CNMF-E, I expected to get an .hdf5 file along with other file outputs so that I could visualize the ROIs as I have before.

Copy paste message from the error window (if any)

Copy-paste or take a screenshot of the terminal that Mesmerize is running in

(mez) scoob@alphaomega:~$ mesmerize
/home/scoob/Desktop/Calcium_Imaging/mez/lib/python3.6/site-packages/sklearn/utils/deprecation.py:143: FutureWarning: The sklearn.cluster.k_means_ module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.cluster. Anything that cannot be imported from sklearn.cluster is now part of the private API.
  warnings.warn(message, FutureWarning)
Loading, please wait... 
%(asctime)s %(levelname)s %(pathname)s %(lineno)s 
 %(message)s 
Moving old logfiles...
Moving old logfiles...
%(asctime)s %(levelname)s %(pathname)s %(lineno)s 
 %(message)s 
EXCEPTION ENCOUNTERED
Traceback (most recent call last):
  File "/home/scoob/Desktop/Calcium_Imaging/mez/lib/python3.6/site-packages/mesmerize/viewer/modules/tiff_io.py", line 99, in check_meta_path
    bn = os.path.basename(self.tiff_file_path)
  File "/usr/lib/python3.6/posixpath.py", line 146, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
EXCEPTION ENCOUNTERED
Traceback (most recent call last):
  File "/home/scoob/Desktop/Calcium_Imaging/mez/lib/python3.6/site-packages/mesmerize/viewer/modules/tiff_io.py", line 99, in check_meta_path
    bn = os.path.basename(self.tiff_file_path)
  File "/usr/lib/python3.6/posixpath.py", line 146, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
%(asctime)s %(levelname)s %(pathname)s %(lineno)s 
 %(message)s 
starting batch mananger
starting batch mananger
showing corr pnr
showing corr pnr
showing corr pnr
showing corr pnr
%(asctime)s %(levelname)s %(pathname)s %(lineno)s 
 %(message)s 
EXCEPTION ENCOUNTERED
Traceback (most recent call last):
  File "/home/scoob/Desktop/Calcium_Imaging/mez/lib/python3.6/site-packages/mesmerize/pyqtgraphCore/exporters/Exporter.py", line 77, in fileSaveFinished
    self.export(fileName=fileName, **self.fileDialog.opts)
  File "/home/scoob/Desktop/Calcium_Imaging/mez/lib/python3.6/site-packages/mesmerize/pyqtgraphCore/exporters/ImageExporter.py", line 70, in export
    bg = np.empty((self.params['width'], self.params['height'], 4), dtype=np.ubyte)
TypeError: 'float' object cannot be interpreted as an integer
EXCEPTION ENCOUNTERED
Traceback (most recent call last):
  File "/home/scoob/Desktop/Calcium_Imaging/mez/lib/python3.6/site-packages/mesmerize/pyqtgraphCore/exporters/Exporter.py", line 77, in fileSaveFinished
    self.export(fileName=fileName, **self.fileDialog.opts)
  File "/home/scoob/Desktop/Calcium_Imaging/mez/lib/python3.6/site-packages/mesmerize/pyqtgraphCore/exporters/ImageExporter.py", line 70, in export
    bg = np.empty((self.params['width'], self.params['height'], 4), dtype=np.ubyte)
TypeError: 'float' object cannot be interpreted as an integer
showing corr pnr

Make sure the entire stack trace is visible, and all messages relevant to the issue are visible.

If relevant, messages from the batch manager terminal status: 0 output_info: Traceback (most recent call last): File "/home/scoob/Desktop/Calcium_Imaging/mez/lib/python3.6/site-packages/mesmerize/viewer/modules/batch_run_modules/CNMFE.py", line 173, in run cnm.fit(Y) File "/home/scoob/Desktop/Calcium_Imaging/CaImAn/caiman/source_extraction/cnmf/cnmf.py", line 606, in fit indices=indices) File "/home/scoob/Desktop/Calcium_Imaging/CaImAn/caiman/source_extraction/cnmf/map_reduce.py", line 249, in run_CNMF_patches file_res = dview.map_async(cnmf_patches, args_in).get(4294967) File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get raise self._value multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f6763dd4cc0>'. Reason: 'PicklingError("Can't pickle <class '_flapack.error'>: import of module '_flapack' failed",)'

Y.shape: [565, 512, 512] processing_time: 1.0152677694956462

Screenshots If applicable, add screenshots to help explain your problem. image

Operating System & specs (CPU, RAM etc.). Please complete the following information:

Details about your Mesmerize install

Additional context I did check the corr/PNR for several gSig values before running the CNMF-E. I don't think this caused the error, but I figured it was worth mentioning.

kushalkolar commented 2 years ago

@WhereWolf72 Thanks for posting the issue.

It's unable to find _flapack which is the scipy interface to low-level linear algebra routines. Can you run this in ipython or the mesmerize console in that environment verify that this doesn't work:

from scipy.linalg import _flapack

I think there's two possibilities:

  1. You're missing LAPACK and/or OpenBLAS. You should be able to install them like this:
sudo apt update
sudo apt install liblapack* libopenblas*

After that create a new python env, install cython, numpy and scipy and try from scipy.linalg import _flapack. If it works then install mesmerize into that environment and see if CNMFE works.

  1. If the above doesn't work your CPU is probably too old to support the newer linear algebra routines.
kushalkolar commented 2 years ago

@WhereWolf72 did this work for you?

WhereWolf72 commented 2 years ago

@kushalkolar Thanks for the prompt help! Unfotunately, those options didn't work. I'm guessing it's my dated CPU then that's the problem. Luckily, I just built a new PC two weeks ago and have an empty drive that I'll dual boot linux onto and try again. Stay tuned.

kushalkolar commented 2 years ago

@WhereWolf72 make sure you install the Blas and lapack math libs before making your Python envs.

WhereWolf72 commented 2 years ago

After setting up the dual boot on my new machine, I was able to install Mesmerize and CaImAn. Initially, I ran into the same problem as before (getting the same issue described above). After installing Blas and lapack separately and creating a new venv into which I installed Mezmerize and CaImAn, it seems the issue is fixed with the caveat that I've only run one file so far. Any idea why these packages needed to be installed separately? When I installed them as per your instructions, it even output "0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded." Thanks again for all your help!

kushalkolar commented 2 years ago

Blas and lapack are dependencies for a lot of scientific applications, but a fresh Ubuntu install might not have them.

More info on Blas and lapack: https://stackoverflow.com/questions/17858104/what-is-the-relation-between-blas-lapack-and-atlas