[Bug]: converting Neuralynx Cheetah "6.4.2 Development" data

[aghatpande]

What happened?

I am trying to import Neuralynx acquired data using neuroconv and got the traceback below (see traceback section) Hoping you have insight into this. On the Neuralynx webpage, the latest version of Cheetah is 6.4.2. The data was acquired using Cheetah "6.4.2 Development" in a Windows 10/11 environment and my analysis machine is a 2020 Macbook Air running MacOS Ventura 13.5.1 . My Python version is 3.11 and the rest of the conda environment is listed below in the package version section. The '...' in the filepaths are for privacy. Thanks in advance for your attention.

Steps to Reproduce

from datetime import datetime
from dateutil import tz
from pathlib import Path
from neuroconv.datainterfaces import NeuralynxRecordingInterface

# For this data interface we need to pass the folder where the data is
folder_path = f"/Users/.../.../.../data/2022-09-16_14-55-53"
# Change the folder_path to the appropriate location in your system
interface = NeuralynxRecordingInterface(folder_path=folder_path, verbose=False)

# Extract what metadata we can from the source files
metadata = interface.get_metadata()
# session_start_time is required for conversion. If it cannot be inferred
# automatically from the source files you must supply one.
session_start_time = datetime(2022, 9, 16, 14, 55, 53, tzinfo=tz.gettz("US/Eastern"))
metadata["NWBFile"].update(session_start_time=session_start_time)

 # Choose a path for saving the nwb file and run the conversion
nwbfile_path = f"/Users/.../.../.../data/2022-09-16_14-55-53/sep162022_nlx2nwb.nwb"  # This should be something like: "./saved_file.nwb"
interface.run_conversion(nwbfile_path=nwbfile_path, metadata=metadata)

Traceback

Traceback (most recent call last):
  File "/Users/.../.../.../python_scripts/nlx2nwb.py", line 9, in <module>
    interface = NeuralynxRecordingInterface(folder_path=folder_path, verbose=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/.../neuroconv/src/neuroconv/datainterfaces/ecephys/neuralynx/neuralynxdatainterface.py", line 43, in __init__
    super().__init__(
  File "/Users/.../neuroconv/src/neuroconv/datainterfaces/ecephys/baserecordingextractorinterface.py", line 32, in __init__
    self.recording_extractor = self.get_extractor()(**source_data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/.../miniconda3/envs/neuroconv_environment/lib/python3.11/site-packages/spikeinterface/extractors/neoextractors/neuralynx.py", line 33, in __init__
    NeoBaseRecordingExtractor.__init__(
  File "/Users/.../miniconda3/envs/neuroconv_environment/lib/python3.11/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py", line 185, in __init__
    _NeoBaseExtractor.__init__(self, block_index, **neo_kwargs)
  File "/Users/.../miniconda3/envs/neuroconv_environment/lib/python3.11/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py", line 25, in __init__
    self.neo_reader = self.get_neo_io_reader(self.NeoRawIOClass, **neo_kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/.../miniconda3/envs/neuroconv_environment/lib/python3.11/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py", line 64, in get_neo_io_reader
    neo_reader.parse_header()
  File "/Users/.../miniconda3/envs/neuroconv_environment/lib/python3.11/site-packages/neo/rawio/baserawio.py", line 179, in parse_header
    self._parse_header()
  File "/Users/.../miniconda3/envs/neuroconv_environment/lib/python3.11/site-packages/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 324, in _parse_header
    _sigs_memmaps, ncsSegTimestampLimits, section_structure = self.scan_stream_ncs_files(
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/.../miniconda3/envs/neuroconv_environment/lib/python3.11/site-packages/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 766, in scan_stream_ncs_files
    chan_ncs_sections = NcsSectionsFactory.build_for_ncs_file(data, nlxHeader)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/.../miniconda3/envs/neuroconv_environment/lib/python3.11/site-packages/neo/rawio/neuralynxrawio/ncssections.py", line 452, in build_for_ncs_file
    raise TypeError("Unknown Ncs file type from header.")
TypeError: Unknown Ncs file type from header.

Operating System

macOS

Python Executable

Conda

Python Version

3.9

Package Versions

Name Version Build Channel

appdirs 1.4.4 pypi_0 pypi arrow 1.2.3 pypi_0 pypi asciitree 0.3.3 pypi_0 pypi attrs 23.1.0 pypi_0 pypi bidsschematools 0.7.1 pypi_0 pypi blessed 1.20.0 pypi_0 pypi boto3 1.28.44 pypi_0 pypi botocore 1.31.44 pypi_0 pypi bzip2 1.0.8 h620ffc9_4 ca-certificates 2023.08.22 hca03da5_0 certifi 2023.7.22 pypi_0 pypi charset-normalizer 3.2.0 pypi_0 pypi ci-info 0.3.0 pypi_0 pypi click 8.1.7 pypi_0 pypi click-didyoumean 0.3.0 pypi_0 pypi dandi 0.56.0 pypi_0 pypi dandischema 0.8.4 pypi_0 pypi dnspython 2.4.2 pypi_0 pypi email-validator 2.0.0.post2 pypi_0 pypi entrypoints 0.4 pypi_0 pypi etelemetry 0.3.0 pypi_0 pypi fasteners 0.18 pypi_0 pypi fparse 1.20.1 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi fscacher 0.4.0 pypi_0 pypi h5py 3.9.0 pypi_0 pypi hdmf 3.9.0 pypi_0 pypi humanize 4.8.0 pypi_0 pypi idna 3.4 pypi_0 pypi importlib-metadata 6.8.0 pypi_0 pypi interleave 0.2.1 pypi_0 pypi isodate 0.6.1 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi jaraco-classes 3.3.0 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi joblib 1.3.2 pypi_0 pypi jsonpointer 2.4 pypi_0 pypi jsonschema 4.19.0 pypi_0 pypi jsonschema-specifications 2023.7.1 pypi_0 pypi keyring 24.2.0 pypi_0 pypi keyrings-alt 5.0.0 pypi_0 pypi libffi 3.4.4 hca03da5_0 more-itertools 10.1.0 pypi_0 pypi natsort 8.4.0 pypi_0 pypi ncurses 6.4 h313beb8_0 neo 0.12.0 pypi_0 pypi neuroconv 0.4.2 pypi_0 pypi numcodecs 0.11.0 pypi_0 pypi numpy 1.25.2 pypi_0 pypi nwbinspector 0.4.29 pypi_0 pypi openssl 3.0.10 h1a28f6b_2 packaging 23.1 pypi_0 pypi pandas 2.1.0 pypi_0 pypi pip 23.2.1 py311hca03da5_0 probeinterface 0.2.17 pypi_0 pypi psutil 5.9.5 pypi_0 pypi pycryptodomex 3.18.0 pypi_0 pypi pydantic 1.10.12 pypi_0 pypi pynwb 2.5.0 pypi_0 pypi pyout 0.7.3 pypi_0 pypi python 3.11.4 hb885b13_0 python-dateutil 2.8.2 pypi_0 pypi pytz 2023.3.post1 pypi_0 pypi pyyaml 6.0.1 pypi_0 pypi quantities 0.14.1 pypi_0 pypi readline 8.2 h1a28f6b_0 referencing 0.30.2 pypi_0 pypi requests 2.31.0 pypi_0 pypi rfc3339-validator 0.1.4 pypi_0 pypi rfc3987 1.3.8 pypi_0 pypi rpds-py 0.10.2 pypi_0 pypi ruamel-yaml 0.17.32 pypi_0 pypi ruamel-yaml-clib 0.2.7 pypi_0 pypi s3transfer 0.6.2 pypi_0 pypi scipy 1.11.2 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi setuptools 68.0.0 py311hca03da5_0 six 1.16.0 pypi_0 pypi spikeinterface 0.98.2 pypi_0 pypi sqlite 3.41.2 h80987f9_0 tenacity 8.2.3 pypi_0 pypi threadpoolctl 3.2.0 pypi_0 pypi tk 8.6.12 hb8d0fd4_0 tqdm 4.66.1 pypi_0 pypi typing-extensions 4.7.1 pypi_0 pypi tzdata 2023.3 pypi_0 pypi uri-template 1.3.0 pypi_0 pypi urllib3 1.26.16 pypi_0 pypi wcwidth 0.2.6 pypi_0 pypi webcolors 1.13 pypi_0 pypi wheel 0.38.4 py311hca03da5_0 xz 5.4.2 h80987f9_0 zarr 2.16.1 pypi_0 pypi zarr-checksum 0.2.9 pypi_0 pypi zipp 3.16.2 pypi_0 pypi zlib 1.2.13 h5a0b063_0

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you ensured this bug was not already reported?

[CodyCBakerPhD]

Hi there @aghatpande!

I've opened issues upstream for this

The process starts with neo: https://github.com/NeuralEnsemble/python-neo/issues/1328

Once available there it goes to spikeinterface: https://github.com/SpikeInterface/spikeinterface/issues/1974

And once that's through everything should work fine

I know that the folks over at neo would really appreciate if you could make a small snippet (less than 10 MB) of example data so they can setup automated tests for it

[aghatpande]

Thanks @CodyCBakerPhD for opening the upstream issues. I'll make a data snippet and post it in the neo issue early next week after clearing it with the folks who own the data. Thanks for your attention and pointing out the heirarchy.

[aghatpande]

@CodyCBakerPhD I created a short example neuralynx dataset which contains one .ncs file (16.6 MB) and one .ntt file (26kB) along with a file called CheetahLogFile.txt and a sub-folder called ConfigurationLog (72 kB) that contains config files, log files. Is this too much as an example? Also, how do I go about conveying it privately to the neo folks since it does contain some identifiers (IP addresses in the config files). Please advise.

[CodyCBakerPhD]

@samuelgarcia @alejoe91 @JuliaSprenger

How would you prefer this user to share their new data snippet? I'd personally recommend google Drive, if they can provide some secure email addresses for you to share it with

As far as identifying info in the config, that will have to be scrubbed eventually anyway (replaced with dummy values) and wouldn't be useful for anything downstream, so I'd recommend you anonymize that as early in the process if you know how to modify it yourself (are they just text files?)

[aghatpande]

@CodyCBakerPhD, @samuelgarcia, @alejoe91, @JuliaSprenger I have uploaded the data snippet to my google drive (only the .ncs and .ntt files for privacy). Please let me know an appropriate email to share it with.

[alejoe91]

you can use this one: alejoe9187@gmail.com

I'll forward it then to @samuelgarcia and @JuliaSprenger

[aghatpande]

I could successfully import Neuralynx data in the latest version of spike interface. Please close this issue. Thank you.

[CodyCBakerPhD]

Glad to hear!