Standardize EEG Data Model for Streaming

alexcastillo commented 7 years ago

Hi everyone,

I've created this repo with the purpose of discussing a data model standard for streaming EEG. Please review the README.md file and feel free to submit PRs.

cc @aj-ptw @marionleborgne @urish @teonbrooks @andrewheusser

Best,

Alex

andrewjaykeller commented 7 years ago

There is a lot of redundancy with this model. The only things that should be in a sample are dynamic variables i.e.:

{
  "buffer": [ // List of samples
    {
      "timestamp": Date, 
      "data": [ // List of channels
         float, 
         float, 
         float, 
         float
      ]
    }
  ]
}

Having to send these over the wire means we need to be lean as possible. I find the best results when i minimize the amount of redundant data between sample.

I think a header or start byte define what the metrics are:

{
  "metric": String // E.g. EEG, ACCL, etc (optional)
  "source": String // E.g. Muse, OpenBCI, etc (optional)
  "sampleRate": Number // E.g. 250. always in hz (optional)
  "mock": Boolean // E.g. false (optional),
  "topology": [ 'Pz', 'Oz', 'Cz', 'C1']
}

especially with hierarchal topics we can inject and split on these exact metrics.

marionleborgne commented 7 years ago

Hey guys. I'm in the mountains now. I will reply on Sunday with a more detailed answer but in short my recommendation for the data model would be to map it to the LSL model.

Send only one time the rich StreamInfo, serialized to JSON. This contains sampling rate, channel positions, channel data format etc.
Open a stream and the only data that is sent is Steam samples serialized to JSON. In LSL parlance it's : {timestamp :..., data : [..., ...]} For efficiency concerns when transmitting over the network we can bundle the samples in chunks and so the JSON transmitted is:
```
[
{timestamp :..., data : [..., ...]},
...,
{timestamp :..., data : [..., ...]}
]
```

Add the quotes to make it real json (I'm writing from my phone) but that's the gist of it. I can add more details when I'm back.

I don't think we should send anything related to the stream info (like electrode positions, sampling rate etc) with the stream samples. It's just inefficient. The stream info should be query-able on demand.

On Fri, Jun 16, 2017, 10:02 PM AJ Keller notifications@github.com wrote:

There is a lot of redundancy with this model. The only things that should be in a sample are dynamic variables i.e.:

{ "buffer": [ // List of samples { "timestamp": Date, "data": [ // List of channels { "id": String // E.g. FP1, FP2, F3, F4, C3, C4, etc (optional) "value": Number }, ... ] } ] }

Having to send these over the wire means we need to be lean as possible. I find the best results when i minimize the amount of redundant data between sample.

I think a header or start byte define what the metrics are:

{ "metric": String // E.g. EEG, ACCL, etc (optional) "source": String // E.g. Muse, OpenBCI, etc (optional) "sampleRate": Number // E.g. 250. always in hz (optional) "mock": Boolean // E.g. false (optional) }

especially with hierarchal topics we can inject and split on these exact metrics.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeuroJS/eeg-stream-data-model/issues/1#issuecomment-309193789, or mute the thread https://github.com/notifications/unsubscribe-auth/ACX3uy72L3IFgrIY7iCi5pBEs81_H0mBks5sE13HgaJpZM4N9H41 .

marionleborgne commented 7 years ago

Basically what AJ says. Except the streaminfo (what he calls the headers) should respect the LSL convention (really, it's the XDF format they're using)

alexandrebarachant commented 7 years ago

Out of curiosity, why not using LSL ? There is no js implementation but I might be easier to make one rather than creating a new standard from scratch.

alexcastillo commented 7 years ago

Thanks everyone for the feedback!

Sounds like we want to go leaner and use LSL. I'm not familiar with LSL but definitely don't want to reinvent the wheel.

@alexandrebarachant can you point me to an LSL implementation in other languages? Maybe a Python implementation.

Maybe we can port it to JavaScript to make the communication between server/client more efficient.

Best,

Alex

alexcastillo commented 7 years ago

Have anyone used this one?

https://github.com/sccn/labstreaminglayer/tree/master/LSL

alexandrebarachant commented 7 years ago

That's the one. I'm using it extensively.

Now, the typical usecase is streaming on a local network, so that is why I'm asking the question.

marionleborgne commented 7 years ago

I've extended LSL to work outside of a lab network. It's in a private repo right now. I will try to release it next week.

On Sat, Jun 17, 2017, 11:16 AM alexandre barachant notifications@github.com wrote:

That's the one. I'm using it extensively.

Now, the topical usecase is streaming on a local network, so that is why I'm asking the question.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeuroJS/eeg-stream-data-model/issues/1#issuecomment-309231321, or mute the thread https://github.com/notifications/unsubscribe-auth/ACX3u8GxU-9dakYi55EClqynikHfzZAVks5sFBfvgaJpZM4N9H41 .

urish commented 7 years ago

I went ahead and created a small node program to stream Muse 2016 data to LSL:

https://github.com/urish/muse-lsl

Basically, it leverages muse-js over noble (BLE bindings for Node.js) using bleat (emulates Web Bluetooth for muse-js), and then uses ffi to interface with liblsl.

So now I got the data streaming to LSL (I can see it in their Lab Recorder program).

Do you have any good pointers for LSL compatible software that I can use to visualize the EEG data, etc.?

alexcastillo commented 7 years ago

This is awesome, Uri!

Could we separate the lsl part into its own lib that can accept an EEG Observable? That way we could use it for the Muse, OpenBCI, etc.

And of course, let me know how I can help.

andrewjaykeller commented 7 years ago

How do I stream to LSL from OpenBCI shield? Arduino C/C++ compatible.

urish commented 7 years ago

thank Alex!

Basically, the LSL wrapper sits in https://github.com/urish/muse-lsl/blob/master/lsl.js, so it can be quite easily separated. I don't have much experience with LSL though - I first heard about it today :)

alexandrebarachant commented 7 years ago

I also have a muse lsl repo : https://github.com/alexandrebarachant/muse-lsl

It's in Python.

There is a lsl viewer to plot live EEG signal.

There is also something to record P300 data.

More detail here : http://alexandre.barachant.org/blog/2017/02/05/P300-with-muse.html

Le 18 juin 2017 00:53, "Uri Shaked" notifications@github.com a écrit :

thank Alex!

Basically, the LSL wrapper sits in https://github.com/urish/muse- lsl/blob/master/lsl.js, so it can be quite easily separated. I don't have much experience with LSL though - I first heard about it today :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeuroJS/eeg-stream-data-model/issues/1#issuecomment-309245105, or mute the thread https://github.com/notifications/unsubscribe-auth/ACC9thhbny9hXW2ieMZ5uw49GF8qVMqRks5sFFjmgaJpZM4N9H41 .

urish commented 7 years ago

Thank @alexandrebarachant , going to try to feed my muse-lsl into the viewer in yours!

What's the best way to communicate with you? Got some questions regarding how to analyze muse data, seems like you have some practical experience with this

alexandrebarachant commented 7 years ago

@urish you can contact me by email. I'm also hanging around in the NeurotechX slack

urish commented 7 years ago

cool, I just joined it! Ping you there

teonbrooks commented 7 years ago

hey everyone, I'm just now catching up. I'm in Cape Town, which is GMT+2. I'm flying tomorrow evening and landing Tuesday afternoon. I will also poke my head into the slack. So far, I like what I am seeing :)

urish commented 7 years ago

Quick update: node js implementation of muse-lsl can now stream values successfully, tested with lsl-viewer. Thanks for @alexandrebarachant a few bugs were fixed :-)

marionleborgne commented 7 years ago

Hi everyone,

This is the data structure of the cloudbrain pub/sub lib. It extends LSL beyond the lab network but is still using the LSL data structure for ease of use in other applications.

LSL refresher

In the LSL model, you have 3 main objects:

StreamInfo to describe the structure of the samples. (The stream info can be used for XDF file headers)
StreamInlet to consume samples.
StreamOutlet to publish samples.

I the following section, I explain the data model that we have been using in Cloudbrain to extend LSL beyond the lab network. It's basically the same as LSL, except that we serialize objects to JSON so that we can have a uniform data contract between distributed applications.

Data model

You have two main objects that are being passed:

Samples: they contain time-stamped channel values. Used by StreamOutlet/ StreamInlet to send/receive data.
StreamInfo: Describes the sample with attributes like sampling_rate, channel_count , etc..

These 2 objects (Samples and StreamInfo) are being serialized to JSON strings to be passed between different applications.

Samples

StreamInlets and StreamOutlets send and receive samples. A sample has the following data format: {"data": [<float>, ..., <float>], "timestamp": <float> }

You can send and receive chunks of samples. In this case:

{
 "chunk": [
    {"data": [<float>, ..., <float>],  "timestamp": <float> },
    ...
    {"data": [<float>, ..., <float>],  "timestamp": <float> }
  ]
}

StreamInfo

In LSL, StreamInfo can be serialized to XML. In the cloudbrain lib that extends LSL, I added the ability to serialize StreamInfo to JSON since this is how other types of data are being serialized and shared between distributed apps.

{
  "info": {
    "name": <string>,
    "type": <string>,
    "channel_count": <int>,
    "nominal_srate": <float>,
    "channel_format": <string>,
    "source_id": <string>,
    "version": <string>,
    "created_at": <timestamp>,
    "uid": <string>,
    "session_id": <string>,
    "hostname": <string>,
    "v4address": <string>,
    "v4data_port": <int>,
    "v4service_port": <int>,
    "v6address": <string>,
    "v6data_port": <string>,
    "v6service_port": <int>,
    "desc": {
      "channels": {
        "channel": [
          {
            "label": <string>,
            "unit": <string>,
            "type": <string>
          },
           ...
          {
            "label": <string>,
            "unit": <string>,
            "type": <string>
          }
        ]
      },
      "cap": {
        "name": <string>,
        "size": <string>
        "labelscheme": <string>
      },
      "manufacturer": <string>
    }
  }
}

Examples

Buffer of 5 samples. Each sample has 2 channels:

{ 
"chunk": [
{ "data":[ 7.056745022195285, 4.754953953750924], "timestamp": 1497479774194733.0},
{ "data":[ 7.056745022195285, 4.754953953750924], "timestamp": 1497479774194733.0},
{ "data":[ 7.056745022195285, 4.754953953750924], "timestamp": 1497479774194733.0},
{ "data":[ 7.056745022195285, 4.754953953750924], "timestamp": 1497479774194733.0},
{ "data":[ 7.056745022195285, 4.754953953750924], "timestamp": 1497479774194733.0}
]
}

StreamInfo

{
"info": {
"name": "muse",
"type": "EEG",
"channel_count": 8,
"nominal_srate": 0,
"channel_format": "float32",
"source_id": "some_id",
"version": "1.1",
"created_at": "0",
"uid": null,
"session_id": null,
"hostname": null,
"v4address": null,
"v4data_port": 0,
"v4service_port": 0,
"v6address": null,
"v6data_port": 0,
"v6service_port": 0,
"desc": {
  "channels": {
    "channel": [
      {
        "label": "FP1",
        "unit": "millivolts",
        "type": "undefined"
      },
      {
        "label": "FP1",
        "unit": "millivolts",
        "type": "undefined"
      },
      {
        "label": "FP1",
        "unit": "millivolts",
        "type": "undefined"
      },
      {
        "label": "FP1",
        "unit": "millivolts",
        "type": "undefined"
      },
      {
        "label": "FP1",
        "unit": "millivolts",
        "type": "undefined"
      },
      {
        "label": "FP1",
        "unit": "millivolts",
        "type": "undefined"
      },
      {
        "label": "FP1",
        "unit": "millivolts",
        "type": "undefined"
      },
      {
        "label": "FP1",
        "unit": "millivolts",
        "type": "undefined"
      }
    ]
  },
  "cap": {
    "name": "undefined",
    "size": "undefined",
    "labelscheme": "undefined"
  },
  "manufacturer": "Interaxon"
}
}
}

alexcastillo commented 7 years ago

@marionleborgne This is perfect, exactly what I was looking for. I'll create some charts that will take this data format as an input.

@urish What do you think? We could update our work to use this data format.

Thanks, everyone.

marionleborgne commented 7 years ago

@alexcastillo glad you like it. @aj-ptw is also using this data contract when pushing samples to cloudbrain from his wifi shield.

marionleborgne commented 7 years ago

@alexcastillo looking forward to the charts! :+1:

urish commented 7 years ago

Thank you so much @marionleborgne !

I just updated muse-js to emit the sample data in this format, and muse-lsl + angular-muse to consume it.

@alexcastillo - also looking forward to the charts :)

andrewjaykeller commented 7 years ago

How am I supposed to stream from Arduino to LSL? Any ideas here?

alexandrebarachant commented 7 years ago

@aj-ptw : your best chance is to ask LSL people directly. there is a C implementation, but i have no idea if you can compile it in a Arduino library.

marionleborgne commented 7 years ago

@aj-ptw another option, since we are in the process of adding cloudbrain connectivity to the wifi shield, is to publish data over MQTT to cloudbrain and then just start the cloudbrain module that publishes data to LSL.

marionleborgne commented 7 years ago

Haha, first thing that comes up when googling this question :p https://github.com/sccn/labstreaminglayer/issues/52 @aj-ptw beats me to it

marionleborgne commented 7 years ago

@alexcastillo @alexandrebarachant @urish @teonbrooks

Question from @aj-ptw and I: anyone has an objections to sending voltage values as nano volts. The channels format will be int.

See https://github.com/OpenBCI/OpenBCI_WIFI/issues/13#issuecomment-310071063 for background.

alexcastillo commented 7 years ago

That's ok with me. Any chance we can add that detail to the StreamInfo? On Wed, Jun 21, 2017 at 4:32 PM Marion Le Borgne notifications@github.com wrote:

@alexcastillo https://github.com/alexcastillo @alexandrebarachant https://github.com/alexandrebarachant @urish https://github.com/urish @teonbrooks https://github.com/teonbrooks

Question from @aj-ptw https://github.com/aj-ptw and I: anyone has an objections to sending voltage values as nano volts. The channels format will be int.

See OpenBCI/OpenBCI_WIFI#13 (comment) https://github.com/OpenBCI/OpenBCI_WIFI/issues/13#issuecomment-310071063 for background.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeuroJS/eeg-stream-data-model/issues/1#issuecomment-310233591, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIH4mNcXmtVGmGIliMtHvgzbiGzQ8yvks5sGaf1gaJpZM4N9H41 .

marionleborgne commented 7 years ago

Actually, it's already in the stream info ("desc" property). See JSON example.

alexcastillo commented 7 years ago

Perfect. On Wed, Jun 21, 2017 at 7:00 PM Marion Le Borgne notifications@github.com wrote:

Actually, it's already in the stream info ("desc" property). See JSON example.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeuroJS/eeg-stream-data-model/issues/1#issuecomment-310254408, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIH4mCppcCwDdjTJ42LEsRV0tb2A0Soks5sGcqqgaJpZM4N9H41 .

marionleborgne commented 7 years ago

@urish i just looked at your muse-lsl repo. It's awesome! Thanks 👍

marionleborgne commented 7 years ago

@urish it occurred to me that you could actually abstract / pull out the LSL part out of the muse-lsl repo and this would be a generic LSL lib for JS - doesn't have to be tied to muse. For example, OpenEXP (@teonbrooks ) can use it publish event markers to LSL. His app is in JS.

You just created the missing JS LSL lib 😄

alexcastillo commented 7 years ago

That's exactly what I told him :) On Thu, Jun 22, 2017 at 5:15 PM Marion Le Borgne notifications@github.com wrote:

@urish https://github.com/urish it occurred to me that you could actually abstract / pull out the LSL part out of the muse-lsl repo and this would be a generic LSL lib for JS - doesn't have to be tied to muse. For example, OpenEXP (@teonbrooks https://github.com/teonbrooks ) can use it publish event markers to LSL. His app in is JS.

You just created the missing JS LSL lib 😄

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeuroJS/eeg-stream-data-model/issues/1#issuecomment-310536236, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIH4qJ8Iydwkx0zBEMnfzC-vXzbsfJdks5sGwOpgaJpZM4N9H41 .

urish commented 7 years ago

here we go: https://www.npmjs.com/package/node-lsl

marionleborgne commented 7 years ago

This is great @urish. Thanks!

NeuroJS / eeg-stream-data-model