[Feature]: Adding video tracking data for neuralynx (.nvt file)

[JohnStout]

What would you like to see added to NeuroConv?

Current neuroconv NeuralynxRecordingInterface does not support .nvt file addition to the NWB file.

The .nvt files represent video tracking data and contain at least X and Y position data (in pixels) as well as their associated timestamps (synchronized to the neural data)

Is your feature request related to a problem?

No response

Do you have any interest in helping implement the feature?

Yes, but I would need guidance.

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you ensured this bug was not already reported?

[CodyCBakerPhD]

Just a reminder if you could share some small-ish examples with us so we can add direct tests on the data

The format seems simple enough so the interface really shouldn't be too hard

[CodyCBakerPhD]

@JohnStout ping

Just a reminder if you could share some small-ish examples with us so we can add direct tests on the data The format seems simple enough so the interface really shouldn't be too hard

[JohnStout]

Hey Cody,

Here is a link to a session recorded from a rat performing a t-maze task ( https://drive.google.com/drive/folders/1zLTovUMjh5S7RqPh1-JE8RlayMXsPYDJ?usp=share_link). That file has the VT1.nvt and Events.nev variables. The Events variable includes important information like timestamps based on infrared beam breaks, while the VT1 file has X/Y/T video tracking data. It should also have an angles variable for head direction, but I would ignore that because it didn't work that well.

Here is another session that includes Tetrodes, CSC.ncs, Events.nev, and VT1.nvt ( https://drive.google.com/drive/folders/1ifVId1cJF4oui2YJEGx7OT7fh2kcUsfB?usp=share_link )

Thanks, John

On Sun, Sep 17, 2023 at 11:34 PM Cody Baker @.***> wrote:

@JohnStout https://github.com/JohnStout ping

Just a reminder if you could share some small-ish examples with us so we can add direct tests on the data The format seems simple enough so the interface really shouldn't be too hard

— Reply to this email directly, view it on GitHub https://github.com/catalystneuro/neuroconv/issues/524#issuecomment-1722708758, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK7U57RBS3RA4VHEOIUBXF3X266NFANCNFSM6AAAAAA2XSXOCE . You are receiving this because you were mentioned.Message ID: @.***>

[CodyCBakerPhD]

Thanks for sharing!

We'll dig into it and try to figure out a good approach for integration

[bendichter]

It looks like neo covers NVT files: https://neo.readthedocs.io/en/latest/rawiolist.html#neo.rawio.NeuralynxRawIO

[bendichter]

I take that back, nvt is not supported: https://github.com/NeuralEnsemble/python-neo/blob/340b022138994fa4ae1ce7b58fb737111b53a2e3/neo/rawio/neuralynxrawio/neuralynxrawio.py#L78

[bendichter]

The best I have found so far is this MATLAB implementation:

https://www.mathworks.com/matlabcentral/fileexchange/26226-readnvt

[bendichter]

Here's a reader that works on the demo data:

import numpy as np
import os
from typing import Dict, Union, List, Tuple

def read_nvt(filename: str) -> Union[Dict[str, np.ndarray], None]:
    """
    Reads a NeuroLynx NVT file and returns its data.

    Parameters
    ----------
    filename : str
        Path to the NVT file.

    Returns
    -------
    Union[Dict[str, np.ndarray], None]
        Dictionary containing the parsed data if file exists, None otherwise.

    Raises
    ------
    FileNotFoundError
        If the specified file does not exist.
    """

    # Constants for header size and record format
    HEADER_SIZE = 16 * 1024
    RECORD_FORMAT = [
        ("swstx", "uint16"),
        ("swid", "uint16"),
        ("swdata_size", "uint16"),
        ("TimeStamp", "uint64"),
        ("dwPoints", "uint32", 400),
        ("sncrc", "int16"),
        ("Xloc", "int32"),
        ("Yloc", "int32"),
        ("Angle", "int32"),
        ("dntargets", "int32", 50),
    ]

    # Check if file exists
    if not os.path.exists(filename):
        raise FileNotFoundError(f"File {filename} not found.")

    # Reading and parsing data
    with open(filename, 'rb') as file:
        file.seek(HEADER_SIZE)
        dtype = np.dtype(RECORD_FORMAT)
        records = np.fromfile(file, dtype=dtype)
        return {name: records[name].squeeze() for name, *_ in RECORD_FORMAT}

# Example usage:
# data = read_nvt("path_to_your_file.nvt")

[bendichter]

data = read_nvt("/Users/bendichter/Downloads/VT1.nvt")

x = data["Xloc"].astype(float)
x[data["Xloc"] <= 0] = np.nan

y = data["Yloc"].astype(float)
y[data["Yloc"] <= 0] = np.nan

plt.plot(x, y)

[bendichter]

from datetime import datetime
from typing import Dict, Union, List

date_parser = lambda x: datetime.strptime(x, "%Y/%m/%d %H:%M:%S")
list_of_ints = lambda x: [int(v) for v in x.split()]
parse_bool = lambda x: x.lower() == "true"
KEY_PARSERS = {
    "TimeCreated": date_parser,
    "TimeClosed": date_parser,
    "RecordSize": int,
    "IntensityThreshold": list_of_ints,
    "RedThreshold": list_of_ints,
    "GreenThreshold": list_of_ints,
    "BlueThreshold": list_of_ints,
    "Saturation": int,
    "Hue": int,
    "Brightness": int,
    "Contrast": int,
    "Sharpness": int,
    "DirectionOffset": int,
    "Resolution": list_of_ints,
    "CameraDelay": int,
    "EnableFieldEstimation": parse_bool,
    "SamplingFrequency": float
}

def parse_header(filename: str) -> Dict[str, Union[str, datetime, float, int, List[int]]]:
    """Parses a Neuralynx Data File Header and returns it as a dictionary."""

    with open(filename, 'rb') as file:
        out = dict()
        for line, _ in zip(file.readlines(), range(27)):
            line = line.decode()
            if line.startswith("-"):
                key, value = line[1:].split(" ", 1)
                value = value.strip()

                # Use the key-specific parser if available, otherwise use default parsing
                parser = KEY_PARSERS.get(key, lambda x: x)
                out[key] = parser(value)
    return out

header = parse_header("/Users/bendichter/Downloads/VT1.nvt")

{'OriginalFileName': 'C:\\Users\\jstout\\Desktop\\Data 2 Move\\21-48\\2023-05-15_10-35-15\\VT1.nvt',
 'TimeCreated': datetime.datetime(2023, 5, 15, 10, 35, 29),
 'TimeClosed': datetime.datetime(2023, 5, 15, 10, 52, 24),
 'FileType': 'Video',
 'FileVersion': '3.3.0',
 'RecordSize': 1828,
 'CheetahRev': '6.4.1 Development',
 'AcqEntName': 'VT1',
 'VideoFormat': 'NTSC',
 'IntensityThreshold': [1, 135],
 'RedThreshold': [1, 100],
 'GreenThreshold': [1, 100],
 'BlueThreshold': [0, 200],
 'Saturation': -1,
 'Hue': -1,
 'Brightness': -1,
 'Contrast': -1,
 'Sharpness': -1,
 'DirectionOffset': 0,
 'Resolution': [720, 480],
 'CameraDelay': 0,
 'EnableFieldEstimation': False,
 'SamplingFrequency': 29.97}

[JohnStout]

Thank you, Ben/catalyst neuro team. Works like a charm.

-John

On Wed, Sep 27, 2023 at 7:30 PM Ben Dichter @.***> wrote:

data = read_nvt("/Users/bendichter/Downloads/VT1.nvt") x = data["Xloc"].astype(float)x[data["Xloc"] <= 0] = np.nan y = data["Yloc"].astype(float)y[data["Yloc"] <= 0] = np.nan plt.plot(x, y)

[image: download (3)] https://user-images.githubusercontent.com/844306/271137930-674c5858-1fd6-4fd2-947b-374293afdee6.png

— Reply to this email directly, view it on GitHub https://github.com/catalystneuro/neuroconv/issues/524#issuecomment-1738238043, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK7U57UE7YLYMLP6IKPR7N3X4SZHVANCNFSM6AAAAAA2XSXOCE . You are receiving this because you were mentioned.Message ID: @.***>

[CodyCBakerPhD]

@JohnStout Hey there - work is almost completed integrating the .nvt support as a new interface

We had one quick question though and hoped you might know how to help - that 'Angle' field present in the file, do you know what units it is in? Degrees or radians?

[JohnStout]

Hey Cody,

I actually haven't used that variable because our system couldn't resolve the LED colors adequately, but Adam from Neuralynx said it is in degrees! If trying to use that variable from our recordings, you'll probably find it to be quite messy.

Thanks! John

On Wed, Oct 25, 2023 at 2:49 PM Cody Baker @.***> wrote:

@JohnStout https://github.com/JohnStout Hey there - work is almost completed integrating the .nvt support as a new interface

We had one quick question though and hoped you might know how to help - that 'Angle' field present in the file, do you know what units it is in? Degrees or radians?

— Reply to this email directly, view it on GitHub https://github.com/catalystneuro/neuroconv/issues/524#issuecomment-1779857936, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK7U57TKQDKOWMGAZQ57ZGDYBFNKRAVCNFSM6AAAAAA2XSXOCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZZHA2TOOJTGY . You are receiving this because you were mentioned.Message ID: @.***>

[bendichter]

@JohnStout yes, for you it's all 0s. Thanks for the help!

[JohnStout]

Makes sense!

No problem, thank you all. -John

On Thu, Oct 26, 2023 at 11:21 AM Ben Dichter @.***> wrote:

@JohnStout https://github.com/JohnStout yes, for you it's all 0s. Thanks for the help!

— Reply to this email directly, view it on GitHub https://github.com/catalystneuro/neuroconv/issues/524#issuecomment-1781341723, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK7U57VBMWQFT4UGUXVRMKTYBJ5WDAVCNFSM6AAAAAA2XSXOCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRGM2DCNZSGM . You are receiving this because you were mentioned.Message ID: @.***>

[JohnStout]

This is probably already handled by your code, but I wanted to bring it up. Because neuralynx collects 512 sample packets of LFP, there are more timestamps than LFP, which requires us to interpolate times to match the size of the LFP signal. If the experimenter hits the "starting recording" and "stopping recording" button (having multiple recording events in a session), they have to interpolate timestamps between those recordings, rather than using the first and last timestamp.

This solution is simple, but I wanted to ask if your code already does this because for students new to collecting data, this issue isn't always obvious. In the figure below, the y-axis represents time (neuralynx time) and the x-axis represents samples as an indexing variable. The black line shows a period of time where no data was collected and was generated by creating two separate timestamp variables (for two separate recordings performed in the same session), then combining them. The blue line was generated by interpolating timestamps using the first and last timestamp. You can see the blue indexing is inaccurate which would lead to inaccuracies when trying to using the timestamps variable as an index for LFP to compare something like behavioral times to LFP.

Anyway, your code may already handle this, but just wanted to mention it because it's important! -John

[image: Screenshot 2023-10-26 at 11.41.38 AM.png]

On Thu, Oct 26, 2023 at 11:32 AM John Stout @.***> wrote:

Makes sense!

No problem, thank you all. -John

On Thu, Oct 26, 2023 at 11:21 AM Ben Dichter @.***> wrote:

@JohnStout https://github.com/JohnStout yes, for you it's all 0s. Thanks for the help!

— Reply to this email directly, view it on GitHub https://github.com/catalystneuro/neuroconv/issues/524#issuecomment-1781341723, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK7U57VBMWQFT4UGUXVRMKTYBJ5WDAVCNFSM6AAAAAA2XSXOCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRGM2DCNZSGM . You are receiving this because you were mentioned.Message ID: @.***>

[bendichter]

John,

Could you please provide the figure in the GitHub issue instead of emailing it?

https://github.com/catalystneuro/neuroconv/issues/524

GitHub isn't able to render images that have been attached in email responses.

Thanks, Ben

[JohnStout]

Ahh, no problem. Here is the figure, uploaded from GitHub.