Issue At Component Evaluation in Demo Pipeline

[neurowookie]

For better support, please use the template below to submit your issue. When your issue gets resolved please remember to close it.

Sometimes errors while running CNMF occur during parallel processing which prevents the log to provide a meaningful error message. Please reproduce your error with setting dview=None.

If you need to upgrade CaImAn follow the instructions given in the documentation.

• Tell us a bit about your setup:

  1. Operating system (Linux/macOS/Windows): Linux
  2. Python version (3.x): 3.7.6
  3. Working environment (Python IDE/Jupyter Notebook/other): Jupyter
  4. Which of the demo scripts you're using for your analysis (if applicable): demo_pipeline
  5. CaImAn version*: 1.8.2
  6. CaImAn installation process (pip install ./pip install -e ./conda): conda create -n caiman
     conda activate caiman conda install caiman -c conda-forge *You can get the CaImAn version by creating a params object and then typing params.data['caiman_version']. If the field doesn't exist, type N/A and consider upgrading)

• Describe the issue that you are experiencing I have a perfectly functioning older version of Caiman on my Windows PC but on our new linux server I can't seem to get it working using the conda forge install. Specifically gets stuck at the CNN component evaluation. I tried with dview as None as well but happens either way(I didn't think this was the issue). It appears to be something to do with keras but I am unsure. Thank you for your input.

• Copy error log below cnm2.estimates.evaluate_components(images, cnm2.params, dview=None) (also happens with dview=dview) USING MODEL:/home/voges/caiman_data/model/cnn_model.json

---

AttributeError Traceback (most recent call last)

in 5 # c) each shape passes a CNN based classifier 6 ----> 7 cnm2.estimates.evaluate_components(images, cnm2.params, dview=None) ~/anaconda3/envs/caiman/lib/python3.7/site-packages/caiman/source_extraction/cnmf/estimates.py in evaluate_components(self, imgs, params, dview) 994 thresh_cnn_lowest=opts['cnn_lowest'], 995 r_values_lowest=opts['rval_lowest'], --> 996 min_SNR_reject=opts['SNR_lowest']) 997 self.idx_components = idx_components.astype(int) 998 self.idx_components_bad = idx_components_bad.astype(int) ~/anaconda3/envs/caiman/lib/python3.7/site-packages/caiman/components_evaluation.py in estimate_components_quality_auto(Y, A, C, b, f, YrA, frate, decay_time, gSig, dims, dview, min_SNR, r_values_min, r_values_lowest, Npeaks, use_cnn, thresh_cnn_min, thresh_cnn_lowest, thresh_fitness_delta, min_SNR_reject, gSig_range) 627 min_SNR, min_SNR_reject, 628 thresh_cnn_min, thresh_cnn_lowest, --> 629 use_cnn, gSig_range) 630 631 return idx_components, idx_components_bad, comp_SNR, r_values, cnn_values ~/anaconda3/envs/caiman/lib/python3.7/site-packages/caiman/components_evaluation.py in select_components_from_metrics(A, dims, gSig, r_values, comp_SNR, r_values_min, r_values_lowest, min_SNR, min_SNR_reject, thresh_cnn_min, thresh_cnn_lowest, use_cnn, gSig_range, neuron_class, predictions, **kwargs) 666 if gSig_range is None: 667 if predictions is None: --> 668 predictions, _ = evaluate_components_CNN(A, dims, gSig) 669 predictions = predictions[:, neuron_class] 670 else: ~/anaconda3/envs/caiman/lib/python3.7/site-packages/caiman/components_evaluation.py in evaluate_components_CNN(A, dims, gSig, model_name, patch_size, loaded_model, isGPU) 305 loaded_model_json = json_file.read() 306 --> 307 loaded_model = model_from_json(loaded_model_json) 308 loaded_model.load_weights(model_name + '.h5') 309 #loaded_model.compile('sgd', 'mse') ~/anaconda3/envs/caiman/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/model_config.py in model_from_json(json_string, custom_objects) 94 config = json.loads(json_string) 95 from tensorflow.python.keras.layers import deserialize # pylint: disable=g-import-not-at-top ---> 96 return deserialize(config, custom_objects=custom_objects) ~/anaconda3/envs/caiman/lib/python3.7/site-packages/tensorflow_core/python/keras/layers/serialization.py in deserialize(config, custom_objects) 104 module_objects=globs, 105 custom_objects=custom_objects, --> 106 printable_module_name='layer') ~/anaconda3/envs/caiman/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name) 290 config = identifier 291 (cls, cls_config) = class_and_config_for_serialized_keras_object( --> 292 config, module_objects, custom_objects, printable_module_name) 293 294 if hasattr(cls, 'from_config'): ~/anaconda3/envs/caiman/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/generic_utils.py in class_and_config_for_serialized_keras_object(config, module_objects, custom_objects, printable_module_name) 252 cls_config = config['config'] 253 deserialized_objects = {} --> 254 for key, item in cls_config.items(): 255 if isinstance(item, dict) and '__passive_serialization__' in item: 256 deserialized_objects[key] = deserialize_keras_object( AttributeError: 'list' object has no attribute 'items'

[j-friedrich]

What is your version of tensorflow (conda list tensorflow inside the caiman environment)? It looks like you are using a newer version of tensorflow.

I updated the json files in the dev branch just yesterday. If you don't want to downgrade to tensorflow 1.15 or 1.13, you can copy https://github.com/flatironinstitute/CaImAn/blob/dev/model/cnn_model.json and https://github.com/flatironinstitute/CaImAn/blob/dev/model/cnn_model_online.json to your_caiman_data_directory/model (these updated files work not only with tensorflow 2, but also already with 1.13+)

[pgunn]

One of the things I tested yesterday before the release was the version of tensorflow you get with a new conda environment; I haven't yet seen a way to end up with anything newer than 1.15 with that initial environment creation.

[j-friedrich]

Using the conda install I ended up with 2.1 on March 5, but maybe there was just a short time window, and it's back to 1.15

[neurowookie]

Ok so for some reason I ended up with 2.1 even though I did install via a new conda env.(If I create another environment it does it again) Copying the cnn_model from dev branch did not help. Downgrading to 1.15 did. Same issue with Bokeh sliders where I had version 2.0 downgrading to 1.3.4 fixed this. Now I am having some sort of issue when restarting the server resulting in a value error in the terminal but I am unsure whether this is related and it doesn't replicate as easily. Should I post this here or open another issue?

[pgunn]

Could you pastebin me the output of these, from your conda environment?:

conda env list

conda list

Thanks

[neurowookie]

conda env list https://pastebin.com/wLazGFBz (caiman2 is me checking whether it installs the newer version of tensorflow again) conda list https://pastebin.com/5wgJVX6U mind I have already downgraded Tensorflow and Bokeh

[pgunn]

Can you pastebin your ~/.condarc ?

[neurowookie]

It is empty?

[pgunn]

I will need to do some experiments to see if I can reproduce the results you're getting - you're ending up with packages mostly from conda-forge, which is surprising to me and not what I'm seeing locally;.

[j-friedrich]

@neurowookie you probably didn't download the raw cnn_model.json. You would do something like wget https://raw.githubusercontent.com/flatironinstitute/CaImAn/dev/model/cnn_model.json or open the mentioned webpage in a browser and do right-click save-page-as. I double-checked that this file works indeed with tensorflow 2.1.

@pgunn the dev branch also includes the fix for the bokeh slider issue. Some code has been deprecated and now that bokeh2 finally broke it, I made the required changes. Once the bokeh2-compatible holoviews 1.13 is released, we could make another release ourselves that supports the major tensorflow and bokeh updates. After deleting ~/.condarc I likewise nevertheless still end up with mostly conda-forge packages. If it matters, conda config --describe channel_priority returns 'flexible'

[neurowookie]

@j-friedrich yes you are correct I did it wrong, with the raw model it works with 2.1 but you are also correct that bokeh2 is broken for now with the model and an old Bokeh I manage to get it to work. However restarting the cluster halfway through the script seems to cause some sort of error that shows up in the terminal after which widgets no longer show. I have a silly amount of ram so its not that important but maybe you have an idea. Here is the terminal output pastebin.com/9q5VK7Fd Thanks for the help.

[j-friedrich]

I encountered the error message in the terminal few times, but not consistently enough to know why it happens. Most of the time it doesn't seem to have any effect. Once or twice it resulted in no plots being displayed and I had to close and reopen the notebook.

[epnev]

This is probably addressed with the latest release so I'm closing it.