DDS Network configuration with USB connection & Orin nano & Notebook

[martinerk0]

I want to use notebook in addition to Orin nano with create3, to see data in Rviz for example, for that I need data from robot, i.e. /tf, /odom etc..

So far I have working communication between robot <--> Orin, and Orin <-> notebook, but I can't see any topics from robot on notebook.

I think this is very common setup, that could be useful to have for other people, if not with Jetsons, many will try to use USB cable+ RPi + notebook.

I already tried various FastDDS xml profiles, but there are not lot of info out there. In addition, when using FastDDS the robot stops sending data after few hours, or start to have 100% CPU and stops communicating.

With CycloneDDS, the communication is much better, but still I can't see anything robot related with notebook. I created access point on Orin, to which I connect with notebook.

I had this setup with Turtlebot 2 which worked flawlessly, but it used Serial connection for communication. Funny thing that ROS2 attempts to improve on networking, but now we need to be network experts too 🥹

For completion, Orin is connected to create3 with USB C like this, both Orin and notebook have Ubuntu 22.04 and ROS Humble, on create3 is H.2.3 firmware.

I want to use access point on Orin, since I will move robot to different locations, and will connect with notebook (or anything else) after start of the robot.

Does anyone have working configuration like this and is willing to share xml profiles (and/or network config) which are you using for DDS setup?

[brianabouchard]

Hi @martinerk0! I believe the issue you are having is related to needing to tell Cyclone DDS on the Orin that you want to use multiple network interfaces. The below XML profile has worked well for me on my Orin:

<CycloneDDS>
   <Domain>
     <General>
        <NetworkInterfaceAddress>l4tbr0,wlan0</NetworkInterfaceAddress>
    </General>
   </Domain>
</CycloneDDS>

Then make sure to export CYCLONEDDS_URI=/path/to/the/xml/profile in terminal on the Orin and replace the path with the path to your new file. You can also permanently add this export to your ~/.bashrc if the configuration works for you.

Let me know if that helps!

[martinerk0]

@brianabouchard Are you sure it works just by listing interfaces? I don't want to connect to create3 with wifi, just use usb connection. Do you have a third PC that is connected (by wifi) just to Orin that you can ros2 topic list --no-daemon ?

It seems that data is not bridged between create3 usb interface and wireless interface on Orin.

They had several issues/topics about this:

• used fastrtp with discovery server: https://github.com/iRobotEducation/create3_docs/issues/184
• this is exacly my problem: https://github.com/iRobotEducation/create3_docs/discussions/366

@tokk-nv is author of Jetson documentation, did you manage to list create3 topics using just usb connection with 3rd PC?

[brianabouchard]

Ahhh, I see. I missed that you did not want to connect the Create 3 to Wi-Fi.

The turtlebot folks put up instructions a little while back on how to add an IP route and use the Fast DDS discovery server that might be of some interest.

I'm also investigating other possibilities and will let you know if I discover anything of interest.

[martinerk0]

Okay, I figured it out! ( and learned new networking commands ❤️)

First I switched to FastDDS, since it seems from https://github.com/turtlebot/turtlebot4/issues/26 that cyclone doesn't work.

I followed https://gist.github.com/roni-kreinin/8fbb20cb4603b8eb5e3961167fb22cd4 and used xml profile from Turlebot4.

Then, started discovery server Orin: fastdds discovery -i 0

after this, (and some ros2 daemon stop & start) I could see /tf and other topics from Orin.

On the other PC, same xml profile, but with 10.42.0.1 i.e. IP of Orin.

After this, it didn't work, but in Turtlebot4 repo they add magic env variable: ROS_DISCOVERY_SERVER

So on Orin: export ROS_DISCOVERY_SERVER="127.0.0.1:11811"

and on 3rd PC:

export ROS_DISCOVERY_SERVER="10.42.0.1:11811"

After this, I could see all topics, but still no data, that's probably because fastdds is relaying ROS topics. On PC, ping 192.168.186.2 revealed that I still can't access data on create3.

In TB4 guide they mention that you have to turn on IP forwarding on Orin(RPi), but this was already set to true sysctl net.ipv4.ip_forward

and also correct ip route on PC ( On Orin it is already set by script if you followed the create3 website) sudo ip route add 192.168.186.0/24 via 10.42.0.1

Last thing I could think of is that l4tbr0 interface doesn't allow traffic in, so I set:

sudo iptables -A FORWARD -o l4tbr0 -j ACCEPT sudo iptables -t nat -A POSTROUTING -o l4tbr0 -j MASQUERADE

Aaand after this, it works! 🎉

I think this is crazy amount of steps to get it working, thanks @brianabouchard for pointing out the Turtlebot repo.

I guess this could be put into service like they do on Turtlebot4, I don't think they would want to add another SBC like Orin Nano, or other Jetson, since it uses other interface names, etc, but I don't think it would be that much work, right?

In addition RPi5 is out and that could have different steps (or not) than RPi4.

Should I put this into docs? @alsora @justinIRBT

I think most people with Jetsons want to use it with some 3rd PC and access point (and USB cable to create3).

Maybe the iptables networking should be put into service somehow like they do in Turtlebot, maybe @tokk-nv could look at it since he made the Jetson scripts?

[brianabouchard]

That's fantastic! Huge thank you for posting your results and process. I'm sure this will be helpful to others as well.

For the docs, I wonder if it makes sense to integrate this into the discovery server page as an additional option? I'd also be happy to add this to the networking recommendations page. @shamlian what do you think?

[martinerk0]

@brianabouchard haha I would be more than happy to make PR myself, but I wonder if it fits into Jetson software config more, since it concerns Jetsons only.

The ROS_DISCOVERY_SERVER env var is new thing and they use it in DDS docs and ROS docs too.

Also, I'm using Humble + 22.04 + Jetpack 6 (DP) natively without docker, which isn't mentioned in Jetson software config so maybe I can update it as well too?

[brianabouchard]

@martinerk0 I'm actually in the process of revising the Jetson docs to separate out devices that can only run up to JetPack 4 (Nano) and add options for a native ROS 2 install on JetPack 5 and 6 as that is my preferred way to run things as well :) I will incorporate your feedback here into the PR.

The environmental variable is interesting to me. We found we did not need it when configuring a discovery server with all the devices on Wi-Fi and all clients as super clients because the xml takes care of that. This is described in a single sentence note in the ROS 2 docs. I suspect this has something to do with all devices being on the same network and specifying a discovery server IP and port when we launch the discovery server. But, I'm by no means a networking expert so I'm curious to learn more.

What I'd love to have are docs that detail multiple options for discovery server implementations with notes on device specific interfaces. From recent posts, the most common use cases seem to be the RPi 4 and the Jetson Orin, but I anticipate we'll see higher usage with RPi 5 coming soon.

[martinerk0]

I don't think adding JetPack 4 (Nano) to documentation is wise, Nano was EOLed and Jetpack 4.x supports only Ubuntu 18.04 and therefore not Galactic or Humble.

It would be nice if Turtlebot 4 guys added support for multiple SBC like Orin & RPi5, but from what I understand RPi 5 needs Ubuntu 23.10 due to driver support. It seems there won't be binaries for 23.x

I have RPi5 at home, so I could build Humble from source, but I don't think anyone would do that anyways.

What I really liked about Turtlebot 2 was that you could run it on almost anything. I haven't yet played with TB4 packages on Orin, but I suppose it won't work out of box. Have you tried TB4 on Orin?

For things like mapping & navigation I suppose all the bringup and setup scripts will be necessary, and doing this network config would be easy in comparison.

[brianabouchard]

I agree that there are concerns on the Nano, but we still have users interested in using them for a lower price point option. Yes, JetPack 4 runs Ubuntu 18.04, but we have actually had great success in upgrading it to Ubuntu 20.04 using some great instructions from Q-engineering.

I can't speak for the SBC choice for TB4, but I can confirm that I've only been able to get the RPi 5 to run Ubuntu 23.10. Slowrunner has been using RPi 5 successfully with a docker container for ROS 2. I've been planning to build Humble from source for my RPi 5 though.

No, I haven't tried the TB4 packages on the Orin, but I don't use TB4 packages for mapping and navigation. I build off of the Create 3 Examples repository.

[martinerk0]

Yeah I also used that image, but in the end, 4GB of memory is only useful in headless mode, RPi5 is much better deal now.

You created mapping/navigation just with create3? If so, that's impressive, I would like to see that :)

Anyway, I think we can close this issue as solved, if you make PR with updated Jetson docs, I would like to have a look at that :)