ZeroCM / zcm

Zero Communications and Marshalling
http://zerocm.github.io/zcm/
GNU Lesser General Public License v2.1
237 stars 69 forks source link

Delay in message from publisher to Spy Lite #377

Closed lhcubillos closed 2 years ago

lhcubillos commented 2 years ago

Hi! I've been working with your package for a few months now, and I discovered an issue that I don't know if is due to something on my implementation or not, and would appreciate a lot your help.

I'm working with an Odroid-XU4 (single-board computer) and running a C++ program that reads encoder data from an Arduino through a serial port and then publishes the data to ZCM using the IPC protocol. To evaluate whether the messages are being sent correctly, I mostly use the zcm-spy-lite tool. It works great if I only run that program, but if I run a heavy processing one on top of that (it's a GUI written in Python), then I notice some delay (around 1s) from the moment I move the encoder to the moment I see the changes in the spy-lite tool. I don't think it gets worse as time goes by, and it gets mostly solved when I close the heavy processing program.

I imagine that there is some buffer filling happening somewhere, and that's why I'm getting that delay, but I'm not sure if it is on the serial side or the ZCM side. Do you have any ideas on why this would happen? Could it be related to ZCM? If it is, would changing the protocol of communication to UDP or others improve that? I don't mind losing messages, I just need data as close to real-time as possible.

The GUI subscribes to those encoder messages and displays them after doing some calculations. I've also noticed an enormous delay (>5s) between the moment the message arrives and it gets shown on the screen. I haven't been able to test yet if some other factors are contributing to that delay, but it seems to be related to ZCM as well.

EDIT: it would also be useful to say that I was able to run everything on my computer and it worked great with no noticeable delays anywhere.

Any kind of help would be greatly appreciated.

Thank you,

Luis

jbendes commented 2 years ago

Hey Luis,

I'm sorry to hear you're having issues. The issue you're running into actually sounds really foreign to zcm. Out of the box, ZCM buffers as little as possible - in fact the default buffer size for both send and receive are each 16 messages. How fast are you publishing when you say you're seeing 5 second delays. Are you able to provide instructions on how to replicate it? I tried stress testing my CPU while running spy lite and a simple publisher but I wasn't able to reproduce the behavior you're seeing. Have you tried console logging in your program whenever you publish? That might let you see if the delay is between the printout and the message showing up in zcm spy. I would probably be a bit more suspicious of the delay being before the call to publish, but of course we all hope the bug is in someone else's code ;) I'm available on gitter too to help if you'd like a bit more realtime help.

Best, Jonathan

On Wed, Dec 1, 2021 at 5:10 PM Luis Cubillos @.***> wrote:

Hi! I've been working with your package for a few months now, and I discovered an issue that I don't know if is due to something on my implementation or not, and would appreciate a lot your help.

I'm working with an Odroid-XU4 (single-board computer) and running a C++ program that reads encoder data from an Arduino through a serial port and then publishes the data to ZCM using the IPC protocol. To evaluate whether the messages are being sent correctly, I mostly use the zcm-spy-lite tool. It works great if I only run that program, but if I run a heavy processing one on top of that (it's a GUI written in Python), then I notice some delay (around 1s) from the moment I move the encoder to the moment I see the changes in the spy-lite tool. I don't think it gets worse as time goes by, and it gets mostly solved when I close the heavy processing program.

I imagine that there is some buffer filling happening somewhere, and that's why I'm getting that delay, but I'm not sure if it is on the serial side or the ZCM side. Do you have any ideas on why this would happen? Could it be related to ZCM? If it is, would changing the protocol of communication to UDP or others improve that? I don't mind losing messages, I just need data as close to real-time as possible.

The GUI subscribes to those encoder messages and displays them after doing some calculations. I've also noticed an enormous delay (>5s) between the moment the message arrives and it gets shown on the screen. I haven't been able to test yet if some other factors are contributing to that delay, but it seems to be related to ZCM as well.

Any kind of help would be greatly appreciated.

Thank you,

Luis

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ZeroCM/zcm/issues/377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXKUUD5UB7OZAORHXCTL23UO2MN7ANCNFSM5JFTDH3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- [image: SkySpecs] https://skyspecs.com/

Jonathan Bendes | Co-founder & Chief Innovation Officer O: 734.413.7346 ext 212 www.skyspecs.com 213 W. Liberty Ste. 300 Ann Arbor, MI 48104 @.***,-83.7527499,17z/data=!3m1!4b1!4m5!3m4!1s0x883caff1f63ba927:0xad9e5f037cc95bc4!8m2!3d42.2792742!4d-83.7505612> [image: SkySpecs Twitter] https://twitter.com/skyspecs [image: SkySpecs Linkedin] https://www.linkedin.com/company-beta/5275893/ [image: SkySpecs Vimeo] https://vimeo.com/skyspecs