Error loading zarr minian data

[BrainofZedZed]

Hello,

Thanks for creating this tool. I'm excited to play around with it. My goal's to use it with Minian datasets but I'm encountering an error reading the zarr data. The error happens after cnmfe.py loads the minian folder as data, when it tries to read the first variable (C). I'm a Python novice, and new to zarr, so it might just be user error. Can you help me identify what I'm missing about reading in the zarr data?

Thanks, Zach

• OS: Windows 11
• Local setup: fresh pynapple install, Python 3.9
• Steps to reproduce:

import numpy as np import pandas as pd import pynapple as nap import matplotlib.pyplot as plt

data_directory = r"C:\Users\Zach\Box\Zach_repo\Projects\Remote memory\Miniscope data\PL_TeA cohort1\D1\ZZ087\My_V4_Miniscope" data = nap.load_session(data_directory, 'minian')

Error:

 File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

 File "c:\users\zach\documents\python scripts\work\pynapple_zz.py", line 16, in <module>
    data = nap.load_session(data_directory, 'minian')

 File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\pynapple\io\misc.py", line 69, in load_session
    return Minian(path)

 File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\pynapple\io\cnmfe.py", line 276, in __init__
    self.load_minian(path)

 File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\pynapple\io\cnmfe.py", line 300, in load_minian
    C = data['C.zarr']['C'][:]

File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\zarr\hierarchy.py", line 361, in __getitem__
    raise KeyError(item)

KeyError: 'C.zarr'

screenshot of directory structure:

[gviejo]

Hey Zach,

Thanks for trying pynapple. Just to be sure, is zarr installed? You can test by doing a pip install zarr in your right environment or doing a import zarr in python.

Let me know,

Best

Guillaume

[BrainofZedZed]

Zarr is installed (version 2.11.3) and imports successfully in cnmfe.py. The minian folder is opened as data and reads as a hierarchy.Group type. I'm not sure how to explore the contents of data much further, but calling any of the other variables that should be saved (A.zarr, b.zarr, etc) also reports back a key error. When I call data.keys(), I just get KeysView(<zarr.hierarchy.Group '/' read-only>) back and nothing obvious pops out in the variable explorer.

[gviejo]

Hey Zach,

By default, pynapple will load only the A and the C of a minian dataset because that's what the NWB can contains. So A and C are accessible with C = data.C and A = data.A.

But you can load additional elements with : miniandata = zarr.open(minian_folder, 'r').

I also realized that miniandata is not attached to the class data which could be useful. I will correct it in the next version.

Let me know if it works for you.

[BrainofZedZed]

Hi Guillaume,

Sorry for the long delay. I think I may have made progress on the issue but still haven't fully solved it. Responding to your suggestion, I can't access any data using either of your suggestions. Trying C = data.C gives a KeyError. It leads me to think that the dataset is not being loaded properly. However, I know the data is there, as I can save it into a different format and load it into Matlab.

On a fresh minian analysis, I tried loading it into Pynapple, and got a different error. This time the error was in cnmfe.py ln 298 data = zarr.open(minian_folder, 'r') *** zarr.errors.PathNotFoundError: nothing found at path ''

If I remove the 'r' argument from zarr.open,
data = zarr.open(minian_folder) I'm able to read the folder (but not access any data), and I get back to the same error as the previous post. Prior to doing that, there is no .zgroup file in the minian folder. After I open the folder without the 'r' argument, the .zgroup file appears, and only then can I read the minian folder with the original line data = zarr.open(minian_folder, 'r')

Again, I'm new to the zarr format, but does this mean I'm missing some sort of metadata file that should be there? I tried reanalyzing a dataset with a fresh install of minian but got back to the same problem. I also tried a fresh install of Pynapple, just in case.

[gviejo]

Hey Zach, I am not sure I understand your problem. Just to be sure, I show you below the different steps to load a minian dataset.

1. In this exemple, I have a folder _demomovies that contains this :

2. Then in python, I give this path:

   import pynapple as nap
   DATA_DIRECTORY = '/home/guillaume/minian/demo_movies'
   data = nap.load_session(DATA_DIRECTORY, 'minian')

3. The GUI shows up to create the NWB file

I pressed ok for both.

4. I have the folder pynapplenwb being created with the NWB file

5. In my ipython console, I can access the data.C and data.A.

Let me know if this helps you. Best

Guillaume

[BrainofZedZed]

Thanks for taking the time to do a thorough step by step. I appreciate the detailed troubleshooting.

I'm able to replicate through step 4 -- I get the NWB popups and create the NWB folder

But then I run into the same error --

Traceback (most recent call last):

  File "C:\Users\Zach\AppData\Local\Temp\ipykernel_16592\4090304052.py", line 1, in <cell line: 1>
    data = nap.load_session(data_directory, 'minian')

  File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\pynapple\io\misc.py", line 69, in load_session
    return Minian(path)

  File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\pynapple\io\cnmfe.py", line 276, in __init__
    self.load_minian(path)

  File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\pynapple\io\cnmfe.py", line 298, in load_minian
    data = zarr.open(minian_folder, 'r')

  File "C:\Users\Zach\anaconda3\envs\pynapple\lib\site-packages\zarr\convenience.py", line 105, in open
    raise PathNotFoundError(path)

PathNotFoundError: nothing found at path '

My minian_folder points to minian output folder but gives a path not found error when opening it with zarr.open(minian_folder, 'r') . What I was trying to say in the previous post is that if I omit the 'r' argument, the command zarr.open(minian_folder) runs, but it doesn't seem like anything is in there.

I've double checked zarr is installed

Would you be willing to see if a sample dataset of mine loads on your setup, to confirm that my minian output is fine? I included a small demo here. Alternatively, do you have a sample dataset I could try loading to test the same?

[gviejo]

Hey Zach, I have the same problem as you. I will try to fix it but I suspect it's a zarr/minian issue or a package compatibility issue.

Here is the dataset I have been using to test the function.

The weird thing is that if you give the direct path to a .zarr folder, it returns the same error. I am really not sure how to address this issue.

[BrainofZedZed]

I'm glad we found the source of it. I agree it's likely a minian/zarr issue. I was able to open and run your dataset just fine. Thank you again for all the troubleshooting (and the package in general)!