How to get multiple subscribers to the same topic (on one machine) to all receive the messages?

[PeterMitrano]

Posting this question here in case anyone here can help. Thanks!

https://robotics.stackexchange.com/questions/110217/lcm-multiple-subscribers-conflict

[ihilt]

This particular use case—a single publisher to multiple subscribers over the network—should work out of the box. Because of the many possible configurations that are possible with LCM, it would be easier to troubleshoot the problem if you provided an example of the code, i.e. the callback handler and publisher.

[PeterMitrano]

Good to hear it's supposed to work, that was my understanding as well. It could be an issue with our network configuration, or with how we're using LCM in our code.

On the sending end, we're using a Java client: https://github.com/UM-ARM-Lab/kuka_iiwa_interface/tree/override_controllers/SunriseWorkspace/UofMR1_2017_RightUpper/src/lcm

On the receiving end we're using C++ client, version 1.3.1 I believe: https://github.com/UM-ARM-Lab/kuka_iiwa_interface/blob/master/lcm/

I am happy to post info about network configuration, but I'm not sure what would be helpful. Here's the output of route on the receiving end. The sending end is Windows so not sure how to do anything there.

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    100    0        0 enp4s0f1
10.10.10.0      0.0.0.0         255.255.255.0   U     100    0        0 enp4s0f1
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 enp4s0f1

As for how the receiving code works, it's not exactly a minimal reproducible example but the actual usage is here: https://github.com/UM-ARM-Lab/kuka_iiwa_interface/blob/override_controllers/victor_hardware/src/side.cpp#L51 https://github.com/UM-ARM-Lab/kuka_iiwa_interface/blob/override_controllers/victor_hardware/include/victor_hardware/lcm_listener.hpp

I'm currently trying to set the TTL, not sure why would help but a coworker suggested it so we'll see...

[ihilt]

My guess is it is not network related because as far as what you've reported, it does work for a single process at least. This proves that the UDP packets are set to the correct TTL such that they can traverse the network interface boundary.

I've only done a cursory glance at the code so I might be way off on this suggestion. It looks like the sending code and listener code are in the same shared library. I wonder if it's possible for the sender and listener, when launched multiple times on the same host, to conflict with one another. My suggestion is this: as a troubleshooting step, disable the code which creates the LCM sender for the build on the subscribing machines and run the receiving test again.

[PeterMitrano]

Thanks, I'll try you suggestion, or otherwise make a small test example that's easier to debug.

One question -- do I need to be using udpm (multcast) in order for this multi-subscriber to work? I thought as long as both subscribers are on the same computer (but possibly different process?) I wouldn't need that. Changing from UDP to UDPM might take a bit more work and it would be nice to avoid changing too many things.

[ihilt]

Thanks, I'll try you suggestion, or otherwise make a small test example that's easier to debug.

A small test example would be great!

One question -- do I need to be using udpm (multcast) in order for this multi-subscriber to work? I thought as long as both subscribers are on the same computer (but possibly different process?) I wouldn't need that. Changing from UDP to UDPM might take a bit more work and it would be nice to avoid changing too many things.

Now I have more questions :smile: What URL are you passing to the lcm_create function? As far as UDP, LCM expects to use UDP multicast addresses for UDP comms, so I'm curious how you're currently setting the subscribers up.

To answer your question though, as far as network communication, LCM supports UDP multicast, multi-port UDP multicast (I haven't tested this so I'm not sure how well it works), and TCP. So, you would need to use one of those in order to pass messages over a network.

Finally, thinking a bit more about your issue, I wonder if the two subscribers are trying to use the same port to receive on. This wouldn't normally be something you'd worry about with UDP multicast but since I'm not sure what sort of network comms you have, this might be something to consider.

[PeterMitrano]

I think you've got it -- we were using UDP (not multicast) because we couldn't get multicast working on the publishing end (which is in Java, on windows, inside our robot's control box...)

provider URLS for sender: https://github.com/UM-ARM-Lab/kuka_iiwa_interface/blob/override_controllers/SunriseWorkspace/UofMR1_2017_RightUpper/src/armlab/lcm/robotInterface/LCMURLs.java provider URLS for receiver: https://github.com/UM-ARM-Lab/kuka_iiwa_interface/blob/override_controllers/victor_hardware/include/victor_hardware/constants.hpp#L11

So I think it actually makes sense that without multicast, two subscribers trying to read on the same UDP port will cause only one of the subscribers to get the data.

[ihilt]

Ah, so the project you've been linking actually has an implementation for LCM UDP, https://github.com/UM-ARM-Lab/kuka_iiwa_interface/blob/override_controllers/lcm/src/lcm/lcm_udp.c. This isn't in the upstream version. That explains my confusion!

So I think it actually makes sense that without multicast, two subscribers trying to read on the same UDP port will cause only one of the subscribers to get the data.

Yeah, if you're doing UDP unicast then this is what I would expect to happen. Possibly changing the java publisher to use UDP broadcast rather than unicast might solve your problem. https://stackoverflow.com/questions/75276132/reading-udp-packets-with-several-clients

[ihilt]

Closing since LCM supports multiple subscribers and the code implementing the UDP subscriber isn't in upstream. Feel free to post again if you run into further issues with LCM.