is it possible to turn off gate_trigger name pending on data files

[babaq]

Hi,

we would like to use SpikeGLX in a remote controlled manner so that our own experiment control software can tell SpikeGLX when to start recording, but the result data file seems always organized in run_gate folder and run_gate_trigger name. we want to have full control of the name and path of the datafile, such as /custom_dir/custom_name.imec.ap.bin.

is there a way right now to achieve this, or would you consider adding this feature? it'll be a great convenience to integrate with our existing environment.

[billkarsh]

There’s a MATLAB command to remotely set the data directory (the parent directory that all run data are written into), so you have control over that and can name it what you want.

You currently cannot skip run folder creation or skip appending ‘_gN_tM’ to file names.

What you ask for, custom naming for every file driven from MATLAB, is pretty specialized, so would not be a high priority for me compared to other features of broader interest. I’m not opposed to this, but I have a number of things going including expanded support for more NI devices. I honestly wouldn’t get to your request for a while.

Could you perhaps rename the files after they are written?

Also, is this a show-stopping issue? Am I going to lose a customer over it?

Bill

From: Li Alex Zhang [mailto:notifications@github.com] Sent: Friday, March 22, 2019 1:54 PM To: billkarsh/SpikeGLX SpikeGLX@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [billkarsh/SpikeGLX] is it possible to turn off gate_trigger name pending on data files (#6)

Hi,

we would like to use SpikeGLX in a remote controlled manner so that our own experiment control software can tell SpikeGLX when to start recording, but the result data file seems always organized in run_gate folder and run_gate_trigger name. we want to have full control of the name and path of the datafile, such as /custom_dir/custom_name.imec.ap.bin.

is there a way right now to achieve this, or would you consider adding this feature? it'll be a great convenience to integrate with our existing environment.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/billkarsh/SpikeGLX/issues/6, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGxB0COI-gKRx_ZwsYqt3LNQqWofPLrJks5vZRjDgaJpZM4cEHTX.

[babaq]

Thanks for the quick reply.

This is definitely not a show-stopping issue, it's just really helpful for us to seamlessly integrate the SpikeGLX data path into our automated data pipeline. but I guess we could rename and move data files.

I know you have a lot of other issue to consider first, hopefully you could enable this feature in the future.

[billkarsh]

I looked at how this might work. I think I can do something next week. See if this spec meets your needs…

I assume this will be used from the MATLAB SDK, so…

(1) I add an SDK function ‘OverrideFileName( string s )’. The string you send is a full path and file base name like this: ‘Y:/…/custom_dir/custom_name’. You send this string prior to triggering file writing.

(2) On the next trigger, if I have received an override string, the next set of files will be named: {s.imeci.ap.bin, s.nidq.meta} and so on. That is, I assume all directories exist, I use your string exactly without adding g/t indices, but I do append the stream identifier and the bin or meta suffix as appropriate.

(3) On using the string, I internally clear it. You have to send me a new string prior to the next trigger, and so on.

Question: Is this for 3A, 3B1 or 3B2?

From: Li Alex Zhang [mailto:notifications@github.com] Sent: Friday, March 22, 2019 1:54 PM To: billkarsh/SpikeGLX SpikeGLX@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [billkarsh/SpikeGLX] is it possible to turn off gate_trigger name pending on data files (#6)

Hi,

we would like to use SpikeGLX in a remote controlled manner so that our own experiment control software can tell SpikeGLX when to start recording, but the result data file seems always organized in run_gate folder and run_gate_trigger name. we want to have full control of the name and path of the datafile, such as /custom_dir/custom_name.imec.ap.bin.

is there a way right now to achieve this, or would you consider adding this feature? it'll be a great convenience to integrate with our existing environment.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/billkarsh/SpikeGLX/issues/6, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGxB0COI-gKRx_ZwsYqt3LNQqWofPLrJks5vZRjDgaJpZM4cEHTX.

[babaq]

from a user perspective, the expected behavior is to be able to set arbitrary data file path:s, and SpikeGLX will only optionally append gate/trigger and definitely append imec.ap.bin/meta to the data path strings.

but if as a addon feature, a new MATLAB SDK function that override the default naming make sense to me. here is my few thoughts:

1, when overriding path strings is received, the directory of the path string s would be better checked and created if not exists. 2, when final data file path is determined as pending imec.ap.bin/meta, this final file path should be checked to avoid overwriting existing file with the same name. 3 then, when any gate/trigger come in, start writing data to the final data path string s.

we have 3A systems, and planning upgrade to 3B2.

[billkarsh]

I am confused.

The point of my auto folder creation and auto g/t tagging is to guarantee that data files are not overwritten, especially when saving trial-based sequences of files. In case it is not clear how that works; every time you click the Enable Recording button, G advances and the trigger program is run again. Every time the trigger program writes a new file-set the T index is advanced.

Your remote controlling program that issues the custom names to me must take responsibility for forming paths and names that make sense. That’s why you want full control, right? If I have to check your names and I decide something is wrong I would have to stop or block the run because only you know how your naming scheme works. You have to be responsible for good names.

I think I am missing a key idea. Help me understand better what you really want. Bill

From: Li Alex Zhang [mailto:notifications@github.com] Sent: Friday, March 22, 2019 3:57 PM To: billkarsh/SpikeGLX SpikeGLX@noreply.github.com Cc: Karsh, Bill karshb@janelia.hhmi.org; Comment comment@noreply.github.com Subject: Re: [billkarsh/SpikeGLX] is it possible to turn off gate_trigger name pending on data files (#6)

from a user perspective, the expected behavior is to be able to set arbitrary data file path:s, and SpikeGLX will only optionally append gate/trigger and definitely append imec.ap.bin/meta to the data path strings.

but if as a addon feature, a new MATLAB SDK function that override the default naming make sense to me. here is my few thoughts:

1, when overriding path strings is received, the directory of the path string s would be better checked and created if not exists. 2, when final data file path is determined as pending imec.ap.bin/meta, this final file path should be checked to avoid overwriting existing file with the same name. 3 then, when any gate/trigger come in, start writing data to the final data path string s.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/billkarsh/SpikeGLX/issues/6#issuecomment-475759894, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGxB0O8kq9K6HVLDIE3NyPX0F4jb5Lapks5vZTWHgaJpZM4cEHTX.

[babaq]

Sorry about confusion, i may not describe the case clearly. here it goes.

Any external program that want to override the SpikeGLX naming convension through remote control should definitely check path and filename conflict, and properly changing test name or increment trial number pending.

What I suggested is just a general consideration that SpikeGLX will put some safeguard on what the external program is asking, so that in case some dangerous thing is asked, such as file overwriting, SpikeGLX can detect it, then it can choose proceed or prompt confirmation or just report error.

Again, it's the external program's responsibility to make correct data path without any conflict, which is exactly our experiment control software does. what I described is like double protection, it's not essential.

[billkarsh]

Is your life better if I give you the MATLAB name override option as I described before, without any safeguards? If so, then I will get it into the next release.

Just to be clear, I propose you send a command from MATLAB with the new name, followed by a command to open the gate and start the triggering (Enable Recording). This will not work with any trigger type except immediate trigger because only that one makes one and only one file-set per gate. If you set any other trigger that makes a sequence of files, your controlling program could not be assured of giving me a new path/name string in time, so file overwriting will occur with the time or TTL triggers. Are you sure this is what you want and expect?

Again, which versions are you using? 3A, 3B1, 3B2? Bill

From: Li Alex Zhang [mailto:notifications@github.com] Sent: Friday, March 22, 2019 5:16 PM To: billkarsh/SpikeGLX SpikeGLX@noreply.github.com Cc: Karsh, Bill karshb@janelia.hhmi.org; Comment comment@noreply.github.com Subject: Re: [billkarsh/SpikeGLX] is it possible to turn off gate_trigger name pending on data files (#6)

Sorry about confusion, i may not describe the case clearly. here it goes.

Any external program that want to override the SpikeGLX naming convension through remote control should definitely check path and filename conflict, and properly changing test name or increment trial number pending.

What I suggested is just a general consideration that SpikeGLX will put some safeguard on what the external program is asking, so that in case some dangerous thing is asked, such as file overwriting, SpikeGLX can detect it, then it can choose proceed or prompt confirmation or just report error.

Again, it's the external program's responsibility to make correct data path without any conflict, which is exactly our experiment control software does. what I described is like double protection, it's not essential.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/billkarsh/SpikeGLX/issues/6#issuecomment-475785886, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGxB0KEaNg99HyVOzmqAzN2WcLU4oEU4ks5vZUf5gaJpZM4cEHTX.

[babaq]

we are currently using 3A, and we could definitely work without SpikeGLX safeguards since we will do it anyway.

what I assumed is the MATLAB override name function will only set the file path, and we could still use TTL triggers to start recording. On another Ripple system we observed that TCP remote command need some time to travel and take effect, so we always waits about 200ms and then send TTL. Is this the timing issue you mention that could make file overwriting. if so, you don't have to worry, we could do proper waiting, so that TTL trigger is always after name overriding.

if it's not, or TTL trigger or trigger as general means automatic tN pending, then we could just use a remote command and immediate triggering as you suggested.

[billkarsh]

Let me amend what I said about how this will work with different triggers…

You will be safe with a triggering event that is under the control of your external program. That could be the immediate or the TTL trigger. What is important is that you need to set the next path&name string in advance of the trigger event. If the controlling program can definitely sequence those you are fine. I admit that at first I imagined your TTL signals would come from an independent periodic source, making you have to race or guess about when you had to give me the next name. That’s because my own TTL testing works off of a pulse generator so I can run for hours.

OK, so I will plan to do this feature next week. Bill

From: Li Alex Zhang [mailto:notifications@github.com] Sent: Friday, March 22, 2019 6:44 PM To: billkarsh/SpikeGLX SpikeGLX@noreply.github.com Cc: Karsh, Bill karshb@janelia.hhmi.org; Comment comment@noreply.github.com Subject: Re: [billkarsh/SpikeGLX] is it possible to turn off gate_trigger name pending on data files (#6)

we are currently using 3A, and we could definitely work without SpikeGLX safeguards since we will do it anyway.

what I assumed is the MATLAB override name function will only set the file path, and we could still use TTL triggers to start recording. On another Ripple system we observed that TCP remote command need some time to travel and take effect, so we always waits about 200ms and then send TTL. Is this the timing issue you mention that could make file overwriting. if so, you don't have to worry, we could do proper waiting, so that TTL trigger is always after name overriding.

if it's not, or TTL trigger or trigger as general means automatic tN pending, then we could just use a remote command and immediate triggering as you suggested.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/billkarsh/SpikeGLX/issues/6#issuecomment-475806223, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGxB0Dn4wtTMgZnYay2fSAZ8RpWRfs6jks5vZVyigaJpZM4cEHTX.

[babaq]

Thanks, looking forward to test it.

[billkarsh]

Feature added to 20190305 as 'SetNextFileName' in MATLAB SDK.