Memory leak in pull_chunk

[GoogleCodeExporter]

What steps will reproduce the problem?

Run this server code, it creates two outlets, one for EEG and one for Marker. 
It randomly populates the EEG outlet, but never touches the Marker outlet.

import random
import time

import pylsl

if __name__ == '__main__':

    info = pylsl.StreamInfo('Test', 'EEG', 8, 100, 'float32', 'myUID')
    data_outlet = pylsl.StreamOutlet(info, 32, 360)

    info = pylsl.StreamInfo('Test', 'Markers', 1, 0, 'string', 'muuid2')
    marker_outlet = pylsl.StreamOutlet(info)

    i = 0
    while True:
        sample = [random.random() for i in range(8)]
        data_outlet.push_sample(sample)
        if i % 10 == 0:
            i = 0
        time.sleep(0.01)

On the client side:

import pylsl
stream = pylsl.resolve_stream('type', 'Markers')
m_inlet = pylsl.StreamInlet(stream[0])
for i in range(10000): m_inlet.pull_chunk(timeout=0)

What is the expected output? What do you see instead?

No increase in memory usage of process. I see 4MB more when ever I call this 
loop.

What version of the product are you using? On what operating system?

Latest from HG.

Please provide any additional information below.

Original issue reported on code.google.com by bastian....@gmail.com on 15 Aug 2014 at 1:36

[GoogleCodeExporter]

I should mention, that the memory leak happens in the client process.

I also found a hint to the problem. If you change the type of the Outlet to 
float32, everything works as expected, so the problem seems to be related to 
the string type.

Original comment by bastian....@gmail.com on 15 Aug 2014 at 2:07

[GoogleCodeExporter]

Same Problem here using the c# wrapper!

Original comment by martin.s...@googlemail.com on 17 Mar 2015 at 12:16

[ulatekh]

The reason seems to be obvious -- lsl_pull_chunk_str() calls malloc() for each string it returns. There's no easy way to call free() from C# (nor from Python, I assume), so an additional C function, e.g. "lsl_chunk_str_free(char *data_buffer, double timestamp_buffer, unsigned long data_buffer_elements, unsigned long data_buffer_row_elements)", would have to be written to call free() in a 2-level loop.

[ulatekh]

It would seem that the C++ interface will have the same problem if an exception is thrown -- lsl::stream_inlet::pull_chunk_multiplexed() and lsl::stream_inlet::pull_sample() both call check_error() (which can throw an exception) before they get a chance to call lsl_destroy_string().

[chkothe]

Hi ulatekh,

good point re string freeing. As you noticed, liblsl actually offers a function just for that purpose, lsl_destroy_string, it's just that the Python interface isn't calling it. We'll address that at the next opportunity.

Now regarding the C++ interface, you're right, though these two functions basically don't throw exceptions in practice unless you disable the recovery option when reading from a stream (in which case you can get one when the sender program disconnects); however, you basically leak one string for each disconnect, and I suppose that would have to happen a few hundred million times in a row until you run out of memory! ;)

Best, Christian

On Thu, Aug 13, 2015 at 1:42 PM, ulatekh notifications@github.com wrote:

It would seem that the C++ interface will have the same problem if an exception is thrown -- lsl::stream_inlet::pull_chunk_multiplexed() and lsl::stream_inlet::pull_sample() both call check_error() (which can throw an exception) before they get a chance to call lsl_destroy_string().

— Reply to this email directly or view it on GitHub https://github.com/sccn/labstreaminglayer/issues/20#issuecomment-130834538 .

[ulatekh]

@chkothe: I've written a fix for the C# interop layer, but haven't been able to test it yet -- I'm still trying to get running with LSL in general. I also fixed some potential memory leaks in lsl_inlet_c.cpp, but again, can't test them yet. I can commit them to a branch of my LSL fork if you want to see them now.

[chkothe]

Hey ulatekh,

thanks, fixes like that are most welcome! Take your time, and feel free to open a pull request on GitHub, that would be most convenient for David and myself! (but patch is also fine)

Best, Christian

On Thu, Aug 13, 2015 at 5:12 PM, ulatekh notifications@github.com wrote:

@chkothe https://github.com/chkothe: I've written a fix for the C# interop layer, but haven't been able to test it yet -- I'm still trying to get running with LSL in general. I also fixed some potential memory leaks in lsl_inlet_c.cpp, but again, can't test them yet. I can commit them to a branch of my LSL fork if you want to see them now.

— Reply to this email directly or view it on GitHub https://github.com/sccn/labstreaminglayer/issues/20#issuecomment-130896811 .

[ulatekh]

https://github.com/sccn/labstreaminglayer/pull/50 contains my fixes for the C# interop layer, as well as some fixes in the existing C/C++ code. Someone in a position to test the python wrapper can use the C# changes as inspiration.