io issue when using on a server. Cannot find h5 files

[milesOIST]

Describe the issue:

Cannot load HDF5 files on the server. Same files load with no problem on the local computer. Checked server suite2p setup using python setup.py test, which gives the following error:

ImportError while importing test module 'suite2p/tests/test_io.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/conda/envs/suite2p/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_io.py:12: in <module>
    from pynwb import NWBHDF5IO
E   ModuleNotFoundError: No module named 'pynwb'
====================================== short test summary info =======================================
ERROR tests/test_io.py

Provide environment info:

     active environment : suite2p
    active env location : /User/conda/envs/suite2p
            shell level : 2
       user config file : /User/.condarc
 populated config files :
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.10.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.28=0
                          __linux=4.18.0=0
                          __unix=0=0
       base environment : /User/conda  (writable)
      conda av data dir : /User/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /User/conda/pkgs
       envs directories :User/conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.28.1 CPython/3.10.10 Linux/4.18.0-348.2.1.el8_5.x86_64 centos/8.5.2111 glibc/2.28
                UID:GID : 2824:9501
             netrc file : None
           offline mode : False

[chriski777]

Hi @milesOIST,

Does your server have pynwb and h5py installed?

[milesOIST]

Hi @chriski777 , I tried reinstalling those packages on the server but no luck. Same error.

[chriski777]

Hi @milesOIST, do the packages show up when you run pip show pynwb and pip show h5py? Sorry, if this seems too pedantic, I just want to make sure that this isn't some server installing packages in the wrong environments/etc.

[milesOIST]

No worries. They're both installed: Name: pynwb Version: 2.3.2 Summary: Package for working with Neurodata stored in the NWB format Home-page: https://github.com/NeurodataWithoutBorders/pynwb Author: Andrew Tritt Author-email: ajtritt@lbl.gov License: BSD

Name: h5py Version: 3.7.0 Summary: Read and write HDF5 files from Python Home-page: http://www.h5py.org Author: Andrew Collette Author-email: andrew.collette@gmail.com License: BSD Location: /hpcshare/appsunit/KuhnU/miles/conda/envs/suite2p/lib/python3.8/site-packages Requires: numpy Required-by: hdmf, pynwb

[chriski777]

Hi @milesOIST, I was able to replicate your issue on a clean environment when following the README.md.

By any chance, are you running pip install -e . before calling python setup.py test? If so, can you run pip install -e .[all] before python setup.py test? We made a recent change to suite2p's dependency such that pynwb wasn't included in the test dependencies (when it should be).

[milesOIST]

It passed all the tests. But still can't find the h5 files. I tried reinstalling using git, then using pip install -e .[all], but still no help. It's odd as suite2p works so well on my desktop.

>         ********************************************************************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        ********************************************************************************

!!
  er_d = dist.fetch_build_eggs(
running egg_info
writing suite2p.egg-info/PKG-INFO
writing dependency_links to suite2p.egg-info/dependency_links.txt
writing entry points to suite2p.egg-info/entry_points.txt
writing requirements to suite2p.egg-info/requires.txt
writing top-level names to suite2p.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
warning: no files found matching 'suite2p/ops/ops_user.npy'
warning: no previously-included files matching '__pycache__' found under directory '*'
adding license file 'LICENSE'
writing manifest file 'suite2p.egg-info/SOURCES.txt'
running build_ext
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.8.16, pytest-7.4.0, pluggy-1.2.0
PyQt5 5.15.9 -- Qt runtime 5.15.2 -- Qt compiled 5.15.2
rootdir: /hpcshare/miles/suite2p
plugins: qt-4.2.0
collected 29 items                                                                                                                                                                                         
tests/test_io.py 
...........                                                                                                                                                                         [ 37%]
tests/test_registration.py ...                                                                                                                                                                       [ 48%]
tests/regression/test_classification_pipeline.py .                                                                                                                                                   [ 51%]
tests/regression/test_detection_pipeline.py ..                                                                                                                                                       [ 58%]
tests/regression/test_extraction_pipeline.py ...                                                                                                                                                     [ 68%]
tests/regression/test_full_pipeline.py ...                                                                                                                                                           [ 79%]
tests/smoke/test_cli.py ....                                                                                                                                                                         [ 93%]
tests/smoke/test_file_loading.py .                                                                                                                                                                   [ 96%]
tests/smoke/test_run_s2p.py .                                                                                                                                                                        [100%]
====================================================================================== 29 passed in 512.61s (0:08:32)

[chriski777]

Can you clarify what you mean by it can't find the H5 files? Are the pathnames to the h5 files not recognized on the clusters?

[milesOIST]

When I run the exact same files on my desktop PC (accessing the files stored on the server), suite2p has no problem finding the h5 files. However, when I try to install suite2p on the server itself, it all installs well. But then I get this message

Could not find any h5 files
>>>ERROR<<<
Traceback (most recent call last):
  File "/server/conda/envs/suite2p/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/server/conda/envs/suite2p/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/server/conda/envs/suite2p/lib/python3.9/site-packages/suite2p/__main__.py", line 84, in <module>
>>>ERROR<<<
    main()
  File "/server/conda/envs/suite2p/lib/python3.9/site-packages/suite2p/__main__.py", line 77, in main
    run_s2p(ops, db)
  File "/server/conda/envs/suite2p/lib/python3.9/site-packages/suite2p/run_s2p.py", line 472, in run_s2p
    ops0 = convert_funs[ops["input_format"]](ops.copy())
  File "/server/conda/envs/suite2p/lib/python3.9/site-packages/suite2p/io/h5.py", line 41, in h5py_to_binary
    ops1, h5list, reg_file, reg_file_chan2 = find_files_open_binaries(ops1, True)
  File "/server/conda/envs/suite2p/lib/python3.9/site-packages/suite2p/io/utils.py", line 231, in find_files_open_binaries
>>>ERROR<<<
    fs, ops2 = get_h5_list(ops1[0])
  File "/server/conda/envs/suite2p/lib/python3.9/site-packages/suite2p/io/utils.py", line 120, in get_h5_list
    raise Exception("no h5s")
Exception: no h5s
Interrupted by error (not finished)

[chriski777]

@milesOIST, it seems the files aren't being found on the server itself. Are the pathnames you're providing in the ops file set correctly with respect to the server?

[chriski777]

Closing this issue due to inactivity. @milesOIST, please feel free to re-open if this issue persists.