search

bonsai-rx / bonsai

The compiler, IDE, and standard library for the Bonsai visual programming language for reactive systems
https://bonsai-rx.org
MIT License
137 stars 29 forks source link

Mismatch between number of samples recorded in independent branches when using AudioPlayback #791

Open hgmarques opened 3 years ago

hgmarques commented 3 years ago

Hi Gonçalo,

I came across with a weird issue while using the OpenEphys for EMG recordings. I have a simple program to extract information from both the head-stage and the (digital) triggers (attached). In my program I use the AudioPlayback node to hear a particular channel. I realized that when using the audio I have a difference in the bytes saved from the trigger and channel data that is not expected. I'm saving each trigger sample with 1 byte and each ephys sample with 64 bytes (sizeof(int16) 32 channels). To give an example, for an ephys file with 2,088,960 bytes I expect a trigger file with 2,088,960 / 64 = 32,640 bytes; however what I get is a trigger file sized 39,360 bytes (files attached WithSound.bin). When I disable the AudioPlayback node the files are as they are supposed to be (files attached *WithoutSound.bin). This offset in the expected number of bytes seems to be consistent.

This to me does not make much sense since I would expect the number of elements going to through the ephys and trigger paths to be exactly the same. In fact, if I count the number of elements (buffers) in each path we get exactly the same number of elements in each. So my questions are:

1) Do you have any idea where the discrepancy comes from?

2) Do you think that it is the trigger file that has extra (garbage) bytes, or is it the ephys file that has missing data? (I seem to be able to read meaningful data in MATLAB, just not sure about the alignment.)

3) Do you think the problem happens at the opening of the files or at closing? (This is important because if you know, I can use that information to align the data I already have.)

Thanks in advance.

Cheers, hugo testephys.zip

glopesdev commented 3 years ago

Hi Hugo,

I’ll need to look at this in more detail, but can you run one more quick test for me?

Try to wrap the AudioPlayback node inside a Sink group. I know this sounds redundant, but I suspect this might have to do with an implementation detail of the AudioPlayback node.

Let me know what happens in this case.

On Mon, 2 Aug 2021 at 13:14, hgmarques @.***> wrote:

Hi Gonçalo,

I came across with a weird issue while using the OpenEphys for EMG recordings. I have a simple program to extract information from both the head-stage and the (digital) triggers (attached). In my program I use the AudioPlayback node to hear a particular channel. I realized that when using the audio I have a difference in the bytes saved from the trigger and channel data that is not expected. I'm saving each trigger sample with 1 byte and each ephys sample with 64 bytes (sizeof(int16) 32 channels). To give an example, for an ephys file with 2,088,960 bytes I expect a trigger file with 2,088,960 / 64 = 32,640 bytes; however what I get is a trigger file sized 39,360 bytes (files attached WithSound.bin). When I disable the AudioPlayback node the files are as they are supposed to be (files attached *WithoutSound.bin). This offset in the expected number of bytes seems to be consistent.

This to me does not make much sense since I would expect the number of elements going to through the ephys and trigger paths to be exactly the same. In fact, if I count the number of elements (buffers) in each path we get exactly the same number of elements in each. So my questions are:

1.

Do you have any idea where the discrepancy comes from? 2.

Do you think that it is the trigger file that has extra (garbage) bytes, or is it the ephys file that has missing data? (I seem to be able to read meaningful data in MATLAB, just not sure about the alignment.) 3.

Do you think the problem happens at the opening of the files or at closing? (This is important because if you know, I can use that information to align the data I already have.)

Thanks in advance.

Cheers, hugo testephys.zip https://github.com/bonsai-rx/bonsai/files/6916714/testephys.zip

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bonsai-rx/bonsai/issues/791, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIR2KEQVA7ZRPQ63BF23D3T22DY7ANCNFSM5BMTSBIQ .

hgmarques commented 3 years ago

Yep. That works. I don't really get it though. How can a node at the end influence the data that comes before it?

glopesdev commented 3 years ago

This can happen because initialization and cleanup of the workflow starts from the end nodes. It is the end nodes that control when subscription to the root source is done. For example, you can use DelaySubscription in one of the branches to trigger a similar loss of synchrony.

If only Transform and Sink nodes are involved, though, there shouldn't be any difference in messages going to both branches. What seems to be happening in this case is the AudioPlayback is a bit special in its initialization, specifically this bit: https://github.com/bonsai-rx/bonsai/blob/565544191a784f17350b6d42127a53ad55899b6f/Bonsai.Audio/AudioPlayback.cs#L67-L69

It is not very obvious, but essentially the AudioPlayback node only subscribes to the input sequence after the AudioSource used to perform the playback is created. In theory this should still be immediate as context initialization seems to be done synchronously, but perhaps there is something inducing the change in behavior and causing the expected behavior of a Sink to be violated.

In any case, whenever something like this happens, it is a valid workaround to use the Sink node to "fix" the behavior, as this group will make sure that anything that is created inside the Sink cannot affect the behavior of the outside workflow.

Will flag this for follow-up.