search

AllenInstitute / AllenSDK

code for reading and processing Allen Institute for Brain Science data
https://allensdk.readthedocs.io/en/latest/
Other
333 stars 149 forks source link

FIlenote FOUnd error for LFP data #2741

Closed AngCamp closed 2 weeks ago

AngCamp commented 5 months ago

My allensdk was working fine untill just 5 minutes ago. THere are two thigns I may have done which may have messed it up.

1) I tried to pip install the mne package in the same environment, to design my own LFP filters. This created conflicts but fortunately I had a .yml of my environmetn saved so I reinstalled it.

2) I interupted a download by interupting the jupyternotebook kernel. I am worried I have messed up my instillation or the cache somehow. I was trying to download the lfp for session 715093703 but now the session is not loading. Suggestions for how to fix this would be helpful.

Problem: The sdk tries to download the session data twice, does so successfully as far as I can tell then it fails to load the data. It can't import the h5l file.

Running... session = cache.get_session_data(session_id)

Returns this error:

WARNING:root:downloading a 2723.916MiB file from http://api.brain-map.org//api/v2/well_known_file_download/1026124469
Downloading: 100%
2.86G/2.86G [00:30<00:00, 109MB/s]
/home/acampbell/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/api/warehouse_cache/caching_utilities.py:127: UserWarning: retrying fetch (1 tries remaining)
  warnings.warn(retry_message)
WARNING:root:downloading a 2723.916MiB file from http://api.brain-map.org//api/v2/well_known_file_download/1026124469
Downloading: 100%
2.86G/2.86G [00:32<00:00, 100MB/s]
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
File [~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/api/warehouse_cache/caching_utilities.py:104](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/api/warehouse_cache/caching_utilities.py:104), in call_caching(fetch, write, read, pre_write, cleanup, lazy, num_tries, failure_message)
    [103](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/api/warehouse_cache/caching_utilities.py:103)         logger.info("Reading data from cache")
--> [104](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/api/warehouse_cache/caching_utilities.py:104)         return read()
    [105](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/api/warehouse_cache/caching_utilities.py:105) except Exception as e:

File [~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:348](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:348), in EcephysProjectCache.get_session_data.<locals>.read(_path)
    [347](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:347) def read(_path):
--> [348](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:348)     session_api = self._build_nwb_api_for_session(_path, session_id, filter_by_validity, **unit_filter_kwargs)
    [349](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:349)     return EcephysSession(api=session_api, test=True)

File [~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:369](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:369), in EcephysProjectCache._build_nwb_api_for_session(self, path, session_id, filter_by_validity, **unit_filter_kwargs)
    [361](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:361) get_analysis_metrics = partial(
    [362](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:362)     self.get_unit_analysis_metrics_for_session,
    [363](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:363)     session_id=session_id,
   (...)
    [366](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:366)     **unit_filter_kwargs
    [367](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:367) )
--> [369](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:369) return EcephysNwbSessionApi(
    [370](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:370)     path=path,
    [371](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:371)     probe_lfp_paths=self._setup_probe_promises(session_id),
    [372](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:372)     additional_unit_metrics=get_analysis_metrics,
    [373](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:373)     external_channel_columns=partial(self._get_substitute_channel_columns, session_id),
    [374](https://vscode-remote+ssh-002dremote-002bitchy-002emsl-002eubc-002eca.vscode-resource.vscode-cdn.net/home/acampbell/Stienmetz2019Reanalyzed/ExtractingSWRs/PowerBandFilters/~/miniconda3/envs/allensdk_env/lib/python3.10/site-packages/allensdk/brain_observatory/ecephys/ecephys_project_cache.py:374)     filter_by_validity=filter_by_validity,
...
File h5py/h5g.pyx:271, in h5py.h5g.GroupID.__init__()

File h5py/h5g.pyx:272, in h5py.h5g.GroupID.__init__()

ImportError: cannot import name h5l
AngCamp commented 5 months ago

I have tried deleting my cache folder entirely and resetting everything by making a new manifest etc, following this tutorial here: https://allensdk.readthedocs.io/en/latest/_static/examples/nb/visual_behavior_neuropixels_LFP_analysis.html

I suspect it it my conda environment so I will try rebuilding it from the ground up without the .yml

AngCamp commented 5 months ago

Ok so I fixed the problem by recreating the environment from scratch. This is a problem however because it means the .yml file I want to use to share the environment is not good enough to recreate my environment complete with other libraries I use in my code. This means my code is not shareable and if I were to try to publish my results would not be reproducible, which is something I want them to be.

So what is the problem with my .yml file, what are some solutions to this issue?

It is as follows: 

name: allensdk_env
channels:
  - edeno
  - bioconda
  - conda-forge
  - r
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - brotli=1.1.0=hd590300_1
  - brotli-bin=1.1.0=hd590300_1
  - bzip2=1.0.8=hd590300_5
  - ca-certificates=2023.11.17=hbcca054_0
  - contourpy=1.2.0=py310hd41b1e2_0
  - cycler=0.12.1=pyhd8ed1ab_0
  - fonttools=4.45.0=py310h2372a71_0
  - freetype=2.12.1=h267a509_2
  - kiwisolver=1.4.5=py310hd41b1e2_1
  - lcms2=2.15=hb7c19ff_3
  - ld_impl_linux-64=2.40=h41732ed_0
  - lerc=4.0.0=h27087fc_0
  - libblas=3.9.0=19_linux64_openblas
  - libbrotlicommon=1.1.0=hd590300_1
  - libbrotlidec=1.1.0=hd590300_1
  - libbrotlienc=1.1.0=hd590300_1
  - libcblas=3.9.0=19_linux64_openblas
  - libdeflate=1.19=hd590300_0
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=13.2.0=h807b86a_2
  - libgfortran-ng=13.2.0=h69a702a_3
  - libgfortran5=13.2.0=ha4646dd_3
  - libgomp=13.2.0=h807b86a_2
  - libjpeg-turbo=3.0.0=hd590300_1
  - liblapack=3.9.0=19_linux64_openblas
  - libnsl=2.0.1=hd590300_0
  - libopenblas=0.3.24=pthreads_h413a1c8_0
  - libpng=1.6.39=h753d276_0
  - libsqlite=3.44.0=h2797004_0
  - libstdcxx-ng=13.2.0=h7e041cc_3
  - libtiff=4.6.0=ha9c0a0a_2
  - libuuid=2.38.1=h0b41bf4_0
  - libwebp-base=1.3.2=hd590300_0
  - libxcb=1.15=h0b41bf4_0
  - libzlib=1.2.13=hd590300_5
  - matplotlib-base=3.8.2=py310h62c0568_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - ncurses=6.4=h59595ed_2
  - openjpeg=2.5.0=h488ebb8_3
  - openssl=3.2.0=hd590300_0
  - packaging=23.2=pyhd8ed1ab_0
  - pip=23.3.1=pyhd8ed1ab_0
  - piso=1.0.0=pyhd8ed1ab_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - pyparsing=3.1.1=pyhd8ed1ab_0
  - python=3.10.2=hc74c709_4_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-tzdata=2023.3=pyhd8ed1ab_0
  - python_abi=3.10=4_cp310
  - pytz=2023.3.post1=pyhd8ed1ab_0
  - readline=8.2=h8228510_1
  - ripple_detection=1.4.0=py_0
  - scipy=1.11.3=py310hb13e2d6_1
  - setuptools=68.2.2=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - sqlite=3.44.0=h2c6b66d_0
  - staircase=2.5.2=pyhd8ed1ab_0
  - tk=8.6.13=noxft_h4845f30_101
  - typing-extensions=4.8.0=hd8ed1ab_0
  - typing_extensions=4.8.0=pyha770c72_0
  - tzdata=2023c=h71feb2d_0
  - unicodedata2=15.1.0=py310h2372a71_0
  - wheel=0.41.3=pyhd8ed1ab_0
  - xorg-libxau=1.0.11=hd590300_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xz=5.2.6=h166bdaf_0
  - zstd=1.5.5=hfc55251_0
  - pip:
    - aiohttp==3.7.4
    - allensdk==2.15.2
    - argschema==3.0.4
    - asttokens==2.4.1
    - async-timeout==3.0.1
    - attrs==23.1.0
    - blosc2==2.3.1
    - boto3==1.17.21
    - botocore==1.20.112
    - cachetools==4.2.4
    - certifi==2023.7.22
    - chardet==3.0.4
    - charset-normalizer==3.3.2
    - comm==0.2.0
    - debugpy==1.8.0
    - decorator==5.1.1
    - distro==1.8.0
    - exceptiongroup==1.1.3
    - executing==2.0.1
    - future==0.18.3
    - glymur==0.8.19
    - greenlet==3.0.1
    - h5py==3.10.0
    - hdmf==3.4.7
    - idna==3.4
    - imageio==2.32.0
    - ipykernel==6.26.0
    - ipython==8.17.2
    - ipywidgets==8.1.1
    - jedi==0.19.1
    - jinja2==3.1.2
    - jmespath==0.10.0
    - jsonschema==4.19.2
    - jsonschema-specifications==2023.7.1
    - jupyter-client==8.6.0
    - jupyter-core==5.5.0
    - jupyterlab-widgets==3.0.9
    - lazy-loader==0.3
    - markupsafe==2.1.3
    - marshmallow==3.20.1
    - matplotlib==3.4.2
    - matplotlib-inline==0.1.6
    - msgpack==1.0.7
    - multidict==6.0.4
    - ndindex==1.7
    - ndx-events==0.2.0
    - nest-asyncio==1.5.8
    - networkx==3.2.1
    - numexpr==2.8.7
    - numpy==1.23.5
    - pandas==1.5.3
    - parso==0.8.3
    - patsy==0.5.3
    - pexpect==4.8.0
    - pillow==10.0.1
    - platformdirs==4.0.0
    - prompt-toolkit==3.0.41
    - psutil==5.9.6
    - psycopg2-binary==2.9.9
    - ptyprocess==0.7.0
    - pure-eval==0.2.2
    - py-cpuinfo==9.0.0
    - pygments==2.17.1
    - pynrrd==0.4.3
    - pynwb==2.2.0
    - pyyaml==6.0.1
    - pyzmq==25.1.1
    - referencing==0.30.2
    - requests==2.31.0
    - requests-toolbelt==0.10.1
    - rpds-py==0.12.0
    - ruamel-yaml==0.18.5
    - ruamel-yaml-clib==0.2.8
    - s3transfer==0.3.7
    - scikit-build==0.17.6
    - scikit-image==0.22.0
    - seaborn==0.13.0
    - semver==3.0.2
    - simpleitk==2.3.1
    - simplejson==3.19.2
    - sqlalchemy==2.0.23
    - stack-data==0.6.3
    - statsmodels==0.14.0
    - tables==3.9.1
    - tifffile==2023.9.26
    - tomli==2.0.1
    - tornado==6.3.3
    - tqdm==4.66.1
    - traitlets==5.13.0
    - urllib3==1.26.18
    - wcwidth==0.2.10
    - widgetsnbextension==4.0.9
    - xarray==2023.10.1
    - yarl==1.9.2
prefix: /home/acampbell/miniconda3/envs/allensdk_env