Pupil Data Epoching/Trialling/Segmentation Library

[gmierz]

Hi,

As part of an undergraduate honors dissertation, I implemented a data segmentation library (called Pupil-Lib) and a server-client pair that can segment raw data obtained from a Pupil-Labs eye tracker based on event markers that are sent from a client and received by the server. Based on the testing I have done, it works very well and the event markers have an error that is about ~10 milliseconds at most which is all that is needed. Further testing needs to be done on some mock data but otherwise, it is ready to be used. It's been tested in two different experiments with a visual stimulus coming from a Matlab instance and in both cases, the library correctly segmented the data without any trouble.

This library's open-source and available here: https://github.com/gmierz/pupil-lib .

Enjoy!

[papr]

Dear @gmierz Thank you for making your project open-source and sharing it with us! Can you elaborate on how you segment the data, i.e. which features you use and what the meaning of these segments is?

[cpicanco]

I was also working on time segmentation based on custom timestamped events triggered during pupil recordings: https://github.com/cpicanco/player_plugins/blob/master/segmentation.py. This idea depends on multi sections: https://github.com/cpicanco/player_plugins/blob/master/trim_marks_patch.py.

Right now it is specific for my own project. I was considering to abstract the idea using also pupil annotations and implementing functionality to read tabular data and functionality to create filters based on data and some user defined parameters (for example, choosing some columns of the data file). With data and filters, one would be able to segment pupil recordings as necessary.

A quick example: During my experiments, I am presenting stimuli. Consider that you need data from the first 2s seconds after stimuli onset. So all you need is a timestamp at each stimuli onset. For example:

timestamp	stimulus
0	A
1	-
2	B
3	-
4	A

and so on

[gmierz]

@cpicanco , it's great to see that we didn't build the exact same thing! :) This library is different in that it's more geared towards use for multi-instrument synchronization experiments like simultaneous EEG and eye-tracking, simultaneous MRI/fMRI and eye-tracking, etc. replacing the more extremely expensive closed-source versions.

That said, I'd love to see the marker functionality you have implemented there added to Pupil Capture so that I could send events from other machines to it and get rid of my server acting as the "middle-man". I wanted to implement that but I just didn't have the time. It's also difficult because I had to run my stimuli in Matlab which meant that I could only use TCP, and not ZMQ (or at least not with little effort).

@papr , a segment is basically a trial (or epoch) that is defined by a marker and a time range that is given to the function 'pupil_epoch'. For example, given a marker at time 5 and a trial range of [-1, 5], we extract from the data a segment which contains all points from 1 second before the marker and up to 5 seconds after the marker. The markers are obtained by running a client on the stimulation machine that sends timestamped markers to the eye-tracker recording machine running the server that then compiles these into a file that we process with 'marker_proc.py' afterwards. The reason for this is because we need to be able to start and stop experiments very quickly and the post-processing takes a bit too long.

That python script creates all the files that are needed by 'pupil_load' to load in a directory of data. The 'pupil_epoch' function is called after and the data is broken down into sets of trials depending on the trigger names. Essentially what I do to segment it is take a marker for a given trigger name, find the timestamped data point that is closest to that marker (improvements to be made in this step), estimate the approximate trial ranges with the sampling rate, minimize the error of the approximation by walking through the timestamps until we get the lowest error, and finally do a linear approximation for an exact trial length (do this for both sides of the trial). Repeat this for all trigger names.

Currently, the only feature I use is the 3D pupil diameter but the library can be easily extended to include any other data (or feature) that is a 2D time series - let me know if this is not what you mean by feature. I only had so much time and pupil diameter was the one we are most interested in for our research.

[cpicanco]

@gmierz ,

it's great to see that we didn't build the exact same thing! :) This library is different in that it's more geared towards use for multi-instrument synchronization experiments like simultaneous EEG and eye-tracking, simultaneous MRI/fMRI and eye-tracking, etc. replacing the more extremely expensive closed-source versions.

I would like to complement by saying that segmentation/marking/multi-sections functionality gives the researcher the ability to filter his/her own data. I guess we solve the exact same problem on this topic (of course, using different strategies and for sure your code is cleaner than mine :smile: ).

To solve multi-instrument synchronization issues, a different problem not related to time segmentation in the previous sense, I choose to run pupil capture in parallel with other software. In the same machine other software uses the same time function, offered by the OS, used by pupil capture). However, right now I do not need mobile/wireless functionality, nor different machines for other instruments, I am good with stationary and a single desktop. I guess multi-instruments that requires different machines should use pupil_synch.

[gmierz]

Ah, ok! Is this a plugin for pupil capture or pupil player?

Actually, I think the LSL plugin that was just recently added would be the best to use for multi-instrument and I will be changing the server I use to that since LSL comes in Matlab also. I found pupil_synch is hard to use with Matlab stimuli and the small increase in precision wasn't worth it.

[cpicanco]

The segmentation plugin is for pupil player. Marking is done in real time using the clock_gettime function.

PS.: If you are reading this and don't know what LSL stands for click here;

[gmierz]

Ah, I see. My version is for pupil capture but I imagine yours works in there - or is easy to get working in there. Thank you for providing the link to LSL!

I've actually just updated the library to work with data obtained from LSL. It was actually very simple and works with the current version of the Pupil Labs LSL Plugin. It just needs one additional outlet stream for markers originating from where ever the stimulation is taking place.

After you stop recording with Lab Recorder, the XDF file it produces can be input into the 'pupil_load' function and processed with 'pupil_epoch'.

[cpicanco]

@gmierz , please don't get me wrong. I am trying to improve the clarity here. So, strictly speaking your library is open source, not free: http://stackoverflow.com/questions/3735389/can-i-license-my-matlab-code-using-the-gpl

[gmierz]

At this point yes you are correct, it's licensed like this Matlab library: EEGLAB: https://sccn.ucsd.edu/wiki/A07:_Contributing_to_EEGLAB

I'll have a python version which will be open-source and free soon, I just have to find a good interpolation function for time series data.

[cpicanco]

I am afraid the license of this matlab library (EEGLAB linked) is not sufficiently clear (at least for me). GPL allows commercial use. Commercial use is different from proprietary/monopoly. Free software (GPL) and also open source software (LGPL) allows commercial use as long as you publishes all the code free of charge. You must publish ALL the code if you redistribute the program/library/ and so on. Matlab code can't be redistributed, it is proprietary.

[gmierz]

I've spoken with some people from EEGLAB and they say there are no problems with making the library available under the GPL license. You should write to them if you questions about it, they would have more information than I can provide.

[cpicanco]

I should note that I was not talking about your matlab code. I was talking about matlab itself, it is a property of mathworks. Also, I would like to say that any code can be released as GPL, even if not all the rules of GPL can be strictly followed in a specific moment in time. For example, today your GPL code depends on non-free matlab code. But in future someone may port your code to a 100% free (as in freedom) environment, using Octave, for example. Of course, porting for freedom may require a lot more work. But this fact does not change the fact that GPL does not prevent commercial use of code released free of charge.

[cpicanco]

From my point of view, your and EEGLAB are doing an excellent work by releasing code as GPL. But you should be aware that GPL does not prevent commercial use.

[gmierz]

Ah, I understand what you are saying now and yes I am aware of that. EEGLAB says that on the page that I link here, but I don't have that written anywhere on my files.

[papr]

Stale

[gmierz]

@cpicanco I have a python version of this library available now: https://github.com/gmierz/pupil-lib-python

It does nearly the same thing except that it's a bit more robust, accurate, extensible, and can be used in a few different ways. I'll only be working on this language version from now on.

How's your segmentation plugin work going?

[cpicanco]

Good! I have stopped working on this specific feature a long time ago. I don't need such a feature for my current work.

On Jan 16, 2018 16:40, "gmierz" notifications@github.com wrote:

@cpicanco https://github.com/cpicanco I have a python version of this library available now: https://github.com/gmierz/pupil-lib-python

It does nearly the same thing except that it's a bit more robust, accurate, extensible, and can be used in a few different ways. I'll only be working on this language version from now on.

How's your segmentation plugin work going?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pupil-labs/pupil/issues/672#issuecomment-358080376, or mute the thread https://github.com/notifications/unsubscribe-auth/AGNc6e9ZeDSNvGsUDNuIXqE6kvcHfagjks5tLPsygaJpZM4Mh3Jj .