question about getting probe from library

[mtehrani-code]

Hello,

I'm following this page: https://probeinterface.readthedocs.io/en/main/examples/ex_10_get_probe_from_library.html to download a probe from the library.

When I type in my probe of interest:

manufacturer = 'neuronexus' probe_name = 'A1x16-5mm-100-177-A16' probe = get_probe(manufacturer, probe_name) print(probe)

I get the error: HTTPError: HTTP Error 404: Not Found

I assume this means that this probe is not available in the library? (it's finding the probe provided in the tutorial: 'A1x32-Poly3-10mm-50-177' so the code must be working). If my probe of interest is not available in the library, what is the next step? Would it be to generate the probe from scratch?

How would this affect the wiring? I use a TDT headstage. Would I need to figure that out manually?

Thanks, M

[zm711]

The docs for creating a probe manually are supplied here. If you generate a map you can also share it with the probeinterface_library for future yes.

And yes you would need to do the wiring manually. I'm not familiar with TDT headstages, but maybe someone else is.

[mtehrani-code]

@zm711 Thank you!

I'm looking at the docs and trying to follow. My probe is a basic 1x16 linear with 100 um site-spacing.

I created a linear probe using:

from probeinterface import generate_linear_probe linear_probe = generate_linear_probe(num_elec=16, ypitch=20) plot_probe(linear_probe, with_contact_id=True)

Then created the json file and edited the probe specs in it:

write_probeinterface('A1x16linearProbe_setup.json', linear_probe)

But when I try to read the file and replot the probe using:

linear_probe = read_probeinterface('A1x16linearProbe_setup.json') plot_probe(linear_probe)

I get the error: KeyError: 'contact_shape_params'

Why the error? Does this mean it's not finding the contact_shape_params?

[mtehrani-code]

Also, once I have the probe and the json file, how do I get the pi.get_probe function to "find" my probe?

I have both files saved locally in my documents. Sorry I didn't see anything about this in the documentations. Only information about how to import probes from the probe library...

[zm711]

Hi Mahtab,

So your first issue is that the representation of probes for probeinterface is actually as a probegroup and not a probe when you use the write_probeinterface function. So when you read the info back in your linear_probe=read_probeinterface(xx) is actually a probegroup. You would need to plot it with plot_probe_group. Alternatively you could do: linear_probe = linear_probe.probes[0] to pull out the single probe from the probegroup.

Also, once I have the probe and the json file, how do I get the pi.get_probe function to "find" my probe?

You have the read_probeinterface function for your homemade probes. So you don't need to do the pi.get_probe you just enter the path

probe_group = pi.read_probeinterface(r'c:\users\...\documents\probe.json')
probe = probe_group.probes[0]

[mtehrani-code]

@zm711 That makes sense! Thank you!

Would it be okay to keep this issue open for a while? I'm working on the manual wiring now and may have questions for you about that. If you prefer that I open a new issue I can do that as well.

-M

[mtehrani-code]

Hi Zach,

I'm working on my probe mapping and running into an indexing error. These are the steps I did:

write_probeinterface('A1x16linearProbe_setup_2.json', linear_probe)

And edited my json file so it looks like this (attached):

A1x16linearProbe_setup_2.json

Then to set up device channel indices I did:

channel_indices = np.arange(16) channel_indices = [6, 11, 3, 14, 1, 16, 2, 15, 5, 12, 4, 13, 7, 10, 8, 9] probe.set_device_channel_indices(channel_indices) print(probe.device_channel_indices)

Here is a map of my probe:

When I type: raw_rec = full_raw_rec.set_probe(probe) raw_rec

I get this error:

---

ValueError Traceback (most recent call last) Cell In[131], line 1 ----> 1 raw_rec = full_raw_rec.set_probe(probe) 2 raw_rec

File ~\Documents\GitHub\spikeinterface\src\spikeinterface\core\baserecordingsnippets.py:96, in BaseRecordingSnippets.set_probe(self, probe, group_mode, in_place) 94 probegroup = ProbeGroup() 95 probegroup.add_probe(probe) ---> 96 return self.set_probes(probegroup, group_mode=group_mode, in_place=in_place)

File ~\Documents\GitHub\spikeinterface\src\spikeinterface\core\baserecordingsnippets.py:170, in BaseRecordingSnippets.set_probes(self, probe_or_probegroup, group_mode, in_place) 163 if number_of_device_channel_indices >= self.get_num_channels(): 164 error_msg = ( 165 f"The given Probe have 'device_channel_indices' that do not match channel count \n" 166 f"{number_of_device_channel_indices} vs {self.get_num_channels()} \n" 167 f"device_channel_indices are the following: {device_channel_indices} \n" 168 f"recording channels are the following: {self.get_channel_ids()} \n" 169 ) --> 170 raise ValueError(error_msg) 172 new_channel_ids = self.get_channel_ids()[device_channel_indices] 173 probe_as_numpy_array = probe_as_numpy_array[order]

ValueError: The given Probe have 'device_channel_indices' that do not match channel count 16 vs 16 device_channel_indices are the following: [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16] recording channels are the following: [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15]

Could you help me figure out what needs to be changed here?

Thank you!

M

[mtehrani-code]

I'll also add that in our set up, the contact IDs match channel numbers: eg. contact # 6 = channel # 6

[samuelgarcia]

Hi, channel_indices must be zeros based (starting form 0 to n-1) here you start from 1 to n.

@zm711 made a very good tutorial during the training day https://github.com/SpikeInterface/SpikeInterface-Training-Edinburgh-May24/tree/main/hands_on/probe_handling https://www.youtube.com/watch?v=pHze_8s4Qak around 17min

[mtehrani-code]

@samuelgarcia Thank you! I watched the video and made some changes. But I'm still getting the same error.

Here is my updated json file: A1x16linearProbe_setup_2.json

and updated code:

channel_indices = np.arange(16) channel_indices = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] probe.set_device_channel_indices(channel_indices) print(probe.device_channel_indices) [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15]

[mtehrani-code]

@zm711 Hi Zach!

I was wondering if you could please check my work on the probe mapping. I changed my channel indices to 0-15 and changed my code to read:

manual_mapping = [5, 10, 2, 13, 0, 15, 1, 14, 4, 11, 3, 12, 6, 9, 7, 8] probe.set_device_channel_indices(manual_mapping)

This seems to have taken care of the "device_channel_indices are the following: [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16] recording channels are the following: [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15] " error.

This is what the table looks like now:

contact_ids shank_ids device_channel_indices 0 6 1 5 1 11 1 10 2 3 1 2 3 14 1 13 4 1 1 0 5 16 1 15 6 2 1 1 7 15 1 14 8 5 1 4 9 12 1 11 10 4 1 3 11 13 1 12 12 7 1 6 13 10 1 9 14 8 1 7 15 9 1 8

and when I type "raw_rec", I get:

BinaryRecordingExtractor: 16 channels - 48.8kHz - 1 segments - 10,546,740 samples 216.12s (3.60 minutes) - float64 dtype - 1.26 GiB file_paths: ['C:\Users\matta\Documents\SI\1500_20230213_01_0_3352_WAV.bin']

However, when I plot the traces as:

%matplotlib widget si.plot_traces(full_raw_rec, backend="ipywidgets", mode='line', channel_ids=raw_rec.channel_ids[0:15])

I get this:

The traces from all channels look identical...so it looks as if it is reading data from one channel only, instead of all 16 channels...Could you help me figure out what's going on?

Thank you!

M

[zm711]

Hey @mtehrani-code,

I recently got back from vacation. Happy to take a look, but bear with me as I get back up to speed.

It is possible that you have some large artifacts across channels. Even if you messed up the mapping however it would just put the channels into different orders not load the same channel. I think we would need to know what recording equipment you're using and why you're loading as a binary file instead of with one of our readers. Could you do the plot traces with:

return_scaled=True and also try filtering your data before plotting it? We can double check your map but I need to see the tdt headstage wiring pattern at the connector to the neuronexus adaptor.

[mtehrani-code]

Hi Zach! Thanks for the update. Hope you enjoyed the time off!

Here are my answers:

Recording equipment: TDT headstage RA16AC-Z

Below are some images of the electrode pinouts on the connector and the headstage:

And this is the electrode contact site layout:

We determined the correct mapping as below:

Regarding our data type:

Our data was generated in MATLAB so based on spikeinterface documentation we decided that our data needs to be converted to binary. We followed the instructions here:

https://spikeinterface.readthedocs.io/en/stable/how_to/load_matlab_data.html?highlight=si.read_binary#loading-data-in-spikeinterface

I get the error below when I try to plot traces with "return_scaled=True":

---

AssertionError Traceback (most recent call last) Cell In[31], line 2 1 get_ipython().run_line_magic('matplotlib', 'widget') ----> 2 si.plot_traces(recording_f, backend="ipywidgets", mode='line', channel_ids=raw_rec.channel_ids[0:15], return_scaled=True)

File ~\Documents\GitHub\spikeinterface\src\spikeinterface\widgets\traces.py:141, in TracesWidget.init(self, recording, segment_index, channel_ids, order_channel_by_depth, time_range, mode, return_scaled, cmap, show_channel_ids, color_groups, color, clim, tile_size, seconds_per_row, with_colorbar, add_legend, backend, **backend_kwargs) 138 mode = mode 139 cmap = cmap --> 141 times, list_traces, frame_range, channel_ids = _get_trace_list( 142 recordings, channel_ids, time_range, segment_index, return_scaled=return_scaled 143 ) 145 # stat for auto scaling done on the first layer 146 traces0 = list_traces[0]

File ~\Documents\GitHub\spikeinterface\src\spikeinterface\widgets\traces.py:553, in _get_trace_list(recordings, channel_ids, time_range, segment_index, return_scaled) 550 fs = rec0.get_sampling_frequency() 552 if return_scaled: --> 553 assert all( 554 rec.has_scaled() for rec in recordings.values() 555 ), "Some recording layers do not have scaled traces. Use return_scaled=False" 556 frame_range = (time_range * fs).astype("int64", copy=False) 557 a_max = rec0.get_num_frames(segment_index=segment_index)

AssertionError: Some recording layers do not have scaled traces. Use return_scaled=False

And this is what the traces look like after I apply filter:

I'm also putting my data file here in case you wanted to look at it directly:

https://drive.google.com/drive/folders/1CuVl_a-gDqYcTBCrB9bCMXG8DOWFk3kY

Thanks,

M

[mtehrani-code]

In spikeinterface, this is what I did for mapping:

channel_indices = np.arange(16) channel_indices = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] probe.set_device_channel_indices(channel_indices)

manual_mapping = [5, 10, 2, 13, 0, 15, 1, 14, 4, 11, 3, 12, 6, 9, 7, 8] probe.set_device_channel_indices(manual_mapping) fig, ax = plt.subplots(figsize=(10,10)) plot_probe(probe, with_contact_id=True, with_device_index=True, ax=ax)

and this is my json file with probe specs currently (only configuration that didn't give me that indexing error...) A1x16linearProbe_setup_2_wired.json

[zm711]

First for your mapping -- to me it looks like

probe - preamplifier channel
1-9
2-10
3-11
4-12
5-13
6-14
7-15
8-16
9-1
10-2

etc. But maybe they don't fit together that way. How did you get your manual mapping?

As far as your data: the error is occurring because you're not supplying the gain and offset to the binary reader. Based on the documentation it looks like a gain of 1, but I don't see documentation for the offset.

In the how_to you link there is a small section explaining the importance of gain and offset for data of this type.

Additionally what format is your data in when it is raw? Does spikeinterface not have a file reader that works to load it ourselves? I might need you to post at least part of your matlab script because if you fed in the wrong data into the binary recording then it is possible all channels would look the same.

[mtehrani-code]

For mapping, the connector and the headstage fit together such that matching numbers on both overlap...seems a bit counterintuitive but that's how they fit.

For manual mapping, I assume you are referring to manual_mapping = [5, 10, 2, 13, 0, 15, 1, 14, 4, 11, 3, 12, 6, 9, 7, 8]?

So this is where I think I need a concept check. Please bear with me. I have the contact sites from 1 to 16. They map onto channels 0-15. But probeinterface uses channel indices which is a way to code the contact sites? So what I need to do is first to assign channel indices to each of the contact IDs, then map the indices to the channel numbers? The indexing must start from zero.

I initially gave each contact site a matching index number: eg. 1 = 1, ...16 =16. Then did the mapping but then got the indexing error. Then I changed the indexing starting at 0, so contact ID 1 = channel index 0, ..., 16 = 15. And on that basis did the mapping:

manual_mapping = [5, 10, 2, 13, 0, 15, 1, 14, 4, 11, 3, 12, 6, 9, 7, 8]

After making this change I did not get the indexing error. But like I said I'm not sure I'm thinking about this correctly.

[mtehrani-code]

As for data, my raw data was in .dat format. I'm not sure if spikeinterface can read a .dat file directly.

I'm putting the relevant scripts that I used to generate the binary file here:

https://drive.google.com/drive/folders/1CuVl_a-gDqYcTBCrB9bCMXG8DOWFk3kY

I hope this helps...I did not write these scripts, that is someone else's job in the lab. But it looks like gain was 1 and offset was 0.

[zm711]

So as long as things fit together correctly then your json file looks right to me. Based on your json file you have a 1500 um probe with spacing of 100 um between contacts. And all of the numbers seem to line up perfectly. Numbers lining up perfectly seems strange to me, but since I don't know how the adaptor and headstage fit together I can 100% confirm that is right. But at least your logic looks right. The goal for the probe mapping is to go from geometry -> contact_id -> device_index -> channel_id. You organized by geometry so I see your mapping was just to get your geometry correctly organized.

.dat is a type of binary file. So SpikeInterface can read that with the read_binary function if you provide it the appropriate info. Let me look at the script. though.

[zm711]

The script you shared is actually missing some of the functions it is calling locally. It seems to be the wrapper script with the actual working occurring in another function. So I can't see what is completely happening. What recording equipment are you using? Basically I'm trying to figure out what/where the header for the .dat is.

[mtehrani-code]

Thanks for confirming that the mapping is correct.

Below is a script (written in matlab but converted to txt so it can be shared here) that demonstrates how our binary data are written, and what data should look like when plotted:

SI_binarydatademo.txt

(The plot is dummy data just as a demo)

[mtehrani-code]

The script you shared is actually missing some of the functions it is calling locally. It seems to be the wrapper script with the actual working occurring in another function. So I can't see what is completely happening. What recording equipment are you using? Basically I'm trying to figure out what/where the header for the .dat is.

To get you all the functions that go into generating the original .dat files I'd have to dig around. I can work on that, but I hope the script I shared above is helpful in the meantime.

As for recording equipment, I'm not sure what you are referring to beyond TDT, neuronexus probe, the RA16AC-Z headstage and MATLAB software...sorry!