liblsl on ESP32-S3

[ValentinPintat]

Hi, We have a project where we want to implement LSL on an ESP32-S3 to create outlets for our sensors. We first try to put micro python on the ESP but we cannot access liblsl with upip. So we thought about putting the C librairies directly build inside our ESP.

Do you have any advice? What would be the closest example?

Thank you very much Valentin

[cboulay]

I'm fairly certain that you'll have to self-compile liblsl, whether you write your app in C or in Python, but I don't know if it's even possible on your platform. You'll need a C-compiler and at least unix-like networking libraries. I took a quick look at ESP-IDF (https://www.espressif.com/en/products/sdks/esp-idf) and it looks like it should have everything you need.

The script at the root of the repo should give you a good starting point: https://github.com/sccn/liblsl/blob/master/standalone_compilation_linux.sh

[ValentinPintat]

Hi, Thank you very much for your answer. We'll try with that starting point then !

[tstenner]

Originally I thought it'd be difficult, because the target platform needs a working environment for Asio, but apparently that's a thing nowadays.

[ValentinPintat]

I don't get it, what'is the link beetween ASIO and liblsl ?

[cboulay]

ASIO is a liblsl dependency. You'll find it in the thirdparty folder.

[dhairyashah1]

I'm working on the project mentioned by @ValentinPintat I included the src, include, thirdparty mentioned here. boost;asio port is there already for esp idf

I was wondering if any other boost file will be required for creating a basic lsl outlet from lslboost

[ValentinPintat]

Is there an easy way to make liblsl available on micropython ?

[cboulay]

"easy" is relative I guess.

1. Build liblsl using that compilation script
2. Install pylsl from source. python -m pip install git+https://github.com/labstreaminglayer/liblsl-Python.git
3. Set an environment variable to tell pylsl where to find liblsl.so built in step 1. Either PYLSL_LIB to the .so file or LD_LIBRARY_PATH to the folder containing the .so file. Alternatively, you can copy the library into a path that's already on the search path, such as in the python package itself.

Step 1 should be possible, but I wouldn't know and I have no way to test.

[ValentinPintat]

Thank you very much for your replies !

Unfortuneltly we don't have those skills in our lab, we're more on the hardware side. But there is still something we want to try ! In the past a student in our Lab achived to implement an LSL outlet on an ESP32 but now all the librairies are dead so we have to redo everything. In his report he said there are 4 stages to start streaming data from an outlet with LSL. (Discovery, Subscription, Data transfer, and Closing). Today we want to try to work on the first phase which is the Discovery and go step by step.

From what I understand, on the discovery phase when the computer is running Labrecorder, an UDP stream is send from Labrecorder to all the network using the host gateway 239.255.172.215 and the port 16571.

So if we re catching that UDP stream and we're sending back with our ESP32 the proper XML info we should see the stream appear on Labrecorder ? Do you know if those informations are still right for the labrecorder that we have today using liblsl V1.16 ?

[ValentinPintat]

Hello ! Here are some news on the project: We manage to get 2 LSL:ShortInfo comming from Labrecorder with our ESP32 on the port 16571.

We can see that Labrecorder is indicating us 2 ports 16572 and 16573.

We know that we have to answer by an XML message but it's not clear for us on what port we have to send the answer ? If we answer properly on the right port will it show as a stream on Labrecorder or will it show as a stream only when we will establish the TCP connection for the Subscritption part ? If we have the right answer will we have an acknolegement from Labrecorder ?

Thank you very much

[cboulay]

I'm very confused. Are you trying to control LabRecorder? Or are you trying to read streams from other devices / applications? Or are you trying to stream data out in a way that LabRecorder will recognize it?

[ValentinPintat]

Sorry about the confusion ! I dont't want to control Labrecorder. I want to do something similar as AudioCapture.exe but embeded in an ESP32. If i'm correct AudioCapture.exe is only creating an Outlet and we can see that outlet available on Labrecorder.

Unfortenely it's very difficult for me to put all the liblsl library on my ESP so instead I want to try to implement an outlet. I managed to create UDP streams, TCP connections and embed XML on my ESP32.

So my question is what are the steps to create an outlet by using TCP, UDP, IGMP and XML. What are the exchange beetween Labrecorder and an outlet ? I want to use Lab recorder to see if my Outlet is visible on the Network, to connect to it and record my data in xdf format to see if it's well transmited.

[cboulay]

The part of the source you should be looking at is here: https://github.com/sccn/liblsl/blob/master/src/stream_outlet_impl.cpp#L14

There's a UDP time server (necessary for clock sync), a UDP multicast responder (necessary to respond to queries to find streams), and finally the data tcp server.

here's how to respond to a query.

here's how to respond to a clock sync.

This method shows you how to initiate the protocol handshake for the TCP server.

IMO the path to getting liblsl compiled on the ESP32 might be shorter than the path to implementing an outlet from scratch, especially if you want it to buffer data, to be robust to network drops, to transmit to multiple inlets, etc.

In any case, I think your development test bench should include:

1. a simple outlet (e.g., from pylsl/examples)
2. a simple script that runs the resolver, makes an inlet from the retrieved StreamInfo, then pulls samples continuously
3. Wireshark
4. A lot of time.

Then, when you've characterized all the packets the simple outlet provides, you can swap it out for your custom implementation on ESP32 until you match the packets 1:1.

[tstenner]

I once wrote a wireshark dissector that splits some LSL packets.

So my question is what are the steps to create an outlet by using TCP, UDP, IGMP and XML. What are the exchange beetween Labrecorder and an outlet ? I want to use Lab recorder to see if my Outlet is visible on the Network, to connect to it and record my data in xdf format to see if it's well transmited.

I would use the liblsl-Python examples (HandleMetadata first, then ReceiveData and ReceiveAndPlot) for that. Anyways, the necessary steps for stream discovery are

• listen for (broadcast / multicast) discovery packets on port 16571.
• match the XPath query against your stream metadata
• if it matches, send a reply packet

[ValentinPintat]

Whaou ! Thank you very much for that !

• listen for (broadcast / multicast) discovery packets on port 16571.
• match the XPath query against your stream metadata
• if it matches, send a reply packet

We manged to try those three steps but we still have some questions. Do we have to send the reply on the port 16571 ? If the reply is correct :

• will we see our "Stream" appearing in green on Labrecorder ?
• will we have an answer from Labrecorder maybe on another port (16572 ?)
• At what step our stream will apprear green on Labrecorder ?

By green I mean that :

[tstenner]

Do we have to send the reply on the port 16571?

The discovery (shortinfo) request has the query in the first line, followed by \r\n and the response port and a query id (source). UDP isn't connection oriented, so the source port for the discovery packet generally isn't the port the reply has to go to.

[thiago-roque07]

Hi!

I'm also working on a project to embed liblsl into an ESP32. @ValentinPintat and @dhairyashah1, would you mind sharing how you did it? I'm planning to use IDF standard C instead of micropython, so technically should be easier. For now I'm getting error with PTHREADS.

`-- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - not found -- Check if compiler accepts -pthread -- Check if compiler accepts -pthread - no CMake Error at /usr/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find Threads (missing: Threads_FOUND) Call Stack (most recent call first): /usr/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE) /usr/share/cmake-3.27/Modules/FindThreads.cmake:226 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) main/lib/liblsl/CMakeLists.txt:167 (find_package)

-- Configuring incomplete, errors occurred! FAILED: build.ninja `

I'm using the Espressif IDF extension to VS Code.

[thiago-roque07]

After a bit of investigation: It seems that Espressif IDF has its own PTHREAD library as part of the IDF framework, and for some reason the "find_package(Threads REQUIRED)" inside liblsl is not finding the pthread library from IDF.

[lucaskdc]

After a bit of investigation: It seems that Espressif IDF has its own PTHREAD library as part of the IDF framework, and for some reason the "find_package(Threads REQUIRED)" inside liblsl is not finding the pthread library from IDF.

Note that the FindThreads (a builtin cmake module) tries to find Threads from the system libraries. I guess that configuring the FindThreads through some cmake variables (CMAKE_THREAD_LIBS_INIT and CMAKE_USE_PTHREADS_INIT) could allow it to find the pthreads from the esp32's lib.

Check this issue regarding Threads being not found on linux: https://github.com/alicevision/geogram/issues/2#issuecomment-536835320

[ValentinPintat]

Hello, we managed to embed lsl on the ESP32-S3. I have to see with the director of my Lab if I can share the code. But I think the student who work on that project kind of "hacked" the system instead of using the regular library. So Basically you have to take care of the exchanges during the TCP and UDP handshake, data transmission, etc....

[thiago-roque07]

@lucaskdc Thank you for the info. I'll try that later.

@ValentinPintat Please, do check if you can share that. I'd love to see that, and it would be of great value for the community!