What steps should I take if I want to run turtlebot3 in ROS2 using an environment I created myself in unreal engine?

[Buddies-as-you-know]

I created my own environment with unreal engine. And there I would like to build a route for turtlebot3 using navigation in ROS2. So here is my question. What steps should I take if I want to run turtlebot3 in ROS2 using an environment I created myself in unreal engine?

[yuokamoto]

I haven't tried but theoritically

1. add RapyutaSimulationPlugins and rclUE as plugin
2. use ROS2GameMode which publish /clock topic and provide ROS 2 services for control Actors
3. place turtlebot3 from editor or spawn from /SpawnEntity ROS2 service.

[Buddies-as-you-know]

There is a repository called turtlebot3-UE, but is it possible to change the environment other than turtlebot (for example, the environment like turtlebot3_benchmark) to other buildings and spaces created by myself with UE5? And is it possible to work with ROS2 navigation?

[yuokamoto]

Yes. you can fork the repository and edit as you like. turtlebot3_benchmark has OccupancyMapGenerator which generate png and yaml files for localization

[james-yoo]

@YuminosukeSato, I'd like to share my hands-on seminar resource on using rclUE and Nav 2 to navigate a custom environment. Although the material is written in Korean, I think you can understand it by looking at the pictures. You can see the Nav 2 integration on pages 22 through 33.

https://docs.google.com/presentation/d/1eZNXPmLtfYkiXLVbxpxOCY--ukxQFXVAd3Uaja0tKRQ/edit#slide=id.g20e9127e323_0_37

[Buddies-as-you-know]

Thank you ! I'll try it.

[Buddies-as-you-know]

I am unable to create an OccupancyGridMap, I get an error when I place the OccupancyGridMap and press the play button. Below is the error.

LoginId:d69c7d530464480cb426a9ddc047c2a3-000003e8
EpicAccountId:

Caught signal 11 Segmentation fault

libUnrealEditor-RapyutaSimulationPlugins.so!_ZN22AOccupancyMapGenerator9BeginPlayEv(+0x20)
libUnrealEditor-Engine.so!AActor::DispatchBeginPlay(bool) [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Engine/Private/Actor.cpp:4053]
libUnrealEditor-Engine.so!AWorldSettings::NotifyBeginPlay() [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Engine/Private/WorldSettings.cpp:291]
libUnrealEditor-Engine.so!AGameMode::HandleMatchHasStarted() [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Engine/Private/GameMode.cpp:221]
libUnrealEditor-Engine.so!AGameMode::SetMatchState(FName) [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Engine/Private/GameMode.cpp:339]
libUnrealEditor-Engine.so!AGameMode::StartPlay() [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Engine/Private/GameMode.cpp:144]
libUnrealEditor-Engine.so!UWorld::BeginPlay() [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Engine/Private/World.cpp:4881]
libUnrealEditor-Engine.so!UGameInstance::StartPlayInEditorGameInstance(ULocalPlayer*, FGameInstancePIEParameters const&) [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Engine/Private/GameInstance.cpp:545]
libUnrealEditor-UnrealEd.so!UEditorEngine::CreateInnerProcessPIEGameInstance(FRequestPlaySessionParams&, FGameInstancePIEParameters const&, int) [/home/iplslam/UnrealEngine/Engine/Source/./Editor/UnrealEd/Private/PlayLevel.cpp:3210]
libUnrealEditor-UnrealEd.so!UEditorEngine::OnLoginPIEComplete_Deferred(int, bool, FString, FPieLoginStruct) [/home/iplslam/UnrealEngine/Engine/Source/./Editor/UnrealEd/Private/PlayLevel.cpp:1672]
libUnrealEditor-UnrealEd.so!UEditorEngine::CreateNewPlayInEditorInstance(FRequestPlaySessionParams&, bool, EPlayNetMode) [/home/iplslam/UnrealEngine/Engine/Source/./Editor/UnrealEd/Private/PlayLevel.cpp:1932]
libUnrealEditor-UnrealEd.so!UEditorEngine::StartPlayInEditorSession(FRequestPlaySessionParams&) [/home/iplslam/UnrealEngine/Engine/Source/./Editor/UnrealEd/Private/PlayLevel.cpp:2937]
libUnrealEditor-UnrealEd.so!UEditorEngine::StartQueuedPlaySessionRequestImpl() [/home/iplslam/UnrealEngine/Engine/Source/./Editor/UnrealEd/Private/PlayLevel.cpp:1237]
libUnrealEditor-UnrealEd.so!UEditorEngine::StartQueuedPlaySessionRequest() [/home/iplslam/UnrealEngine/Engine/Source/./Editor/UnrealEd/Private/PlayLevel.cpp:1149]
libUnrealEditor-UnrealEd.so!UEditorEngine::Tick(float, bool) [/home/iplslam/UnrealEngine/Engine/Source/./Editor/UnrealEd/Private/EditorEngine.cpp:1776]
libUnrealEditor-UnrealEd.so!UUnrealEdEngine::Tick(float, bool) [/home/iplslam/UnrealEngine/Engine/Source/./Editor/UnrealEd/Private/UnrealEdEngine.cpp:514]
UnrealEditor!FEngineLoop::Tick() [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Launch/Private/LaunchEngineLoop.cpp:5363]
UnrealEditor!GuardedMain(char16_t const*) [/home/iplslam/UnrealEngine/Engine/Source/./Runtime/Launch/Private/Launch.cpp:204]
libUnrealEditor-UnixCommonStartup.so!CommonUnixMain(int, char**, int (*)(char16_t const*), void (*)()) [/home/iplslam/UnrealEngine/Engine/Source/Runtime/Unix/UnixCommonStartup/Private/UnixCommonStartup.cpp:269]
libc.so.6!__libc_start_main(+0xf2)
UnrealEditor!_start()

[yuokamoto]

Did you set Map? https://rapyutasimulationplugins.readthedocs.io/en/latest/doxygen_generated/html/d2/dde/class_a_occupancy_map_generator.html#a6052b2bc9de6a01e06f37c2c565edb17

[Buddies-as-you-know]

I didn't set Map. I try to set Map.But I don't get map image. Screencast from 2023年06月17日 17時01分12秒.webm

[yuokamoto]

Please check

• MaxVrticalHeight: https://rapyutasimulationplugins.readthedocs.io/en/latest/doxygen_generated/html/d2/dde/class_a_occupancy_map_generator.html#a4984d0bdf4e3c5347a20f3dbf1c03522
• Collision/Trace setting of wall and obstacle and bounding box of your Map. It uses tracing to generate occupancy map and that size is decide based on Map bounding box. https://github.com/rapyuta-robotics/RapyutaSimulationPlugins/blob/8ff56736c0026f53f828075ece9af456e40e8f19/Source/RapyutaSimulationPlugins/Private/Tools/OccupancyMapGenerator.cpp#LL47C21-L47C38

[Buddies-as-you-know]

The connection is made and the map is generated, but the waffle moves in the opposite direction of navigation. Screencast from 2023年06月19日 14時31分10秒.webm

[Buddies-as-you-know]

I have one more question: we have attached a Yolo camera, but the camera does not work with waffle. How can I get it to work with turtolbot3?Screencast from 2023年06月19日 15時25分49秒.webm

[yuokamoto]

Cool

• Did you check waffle move properly with cmd_vel?
• It is better to start with Turtlbot3WaffleVehicle, i.e. kinematic model since physics requires many parameter tunings
• but the camera does not work with waffle, do you mean came do not move with tb3? did you check Mobility of camera?

[Buddies-as-you-know]

Did you check waffle move properly with cmd_vel?　-> No. I don't know way to check waffle move properly with cmd_vel. It is better to start with Turtlbot3WaffleVehicle, i.e. kinematic model since physics requires many parameter tunings->ok.I try it. but the camera does not work with waffle, do you mean came do not move with tb3? -> Yes did you check Mobility of camera?-> How can I check the mobility of my camera?

[Buddies-as-you-know]

Only the Turtlbot3WaffleVehicle does not land on the floor and falls. How can I prevent it from falling during execution? Screencast from 2023年06月21日 14時57分49秒.webm

[james-yoo]

@YuminosukeSato You did not generate collision to static mesh(in this case floor).

[Buddies-as-you-know]

The Turtlbot3BurgurVehicle lands firmly on the floor, but the Turtlbot3WaffleVehicle does not.

[yuokamoto]

Please check collision setting of floor and obstacles

• Please check turltbot3-UE map's floor collision setting and try to set simillar setting to your static meshes. ref https://docs.unrealengine.com/4.27/ja/InteractiveExperiences/Physics/Collision/Reference/

• You can visualize collision at runtime by https://docs.unrealengine.com/4.27/ja/InteractiveExperiences/Physics/Collision/HowTo/PreviewCollision/

• about mobility https://docs.unrealengine.com/4.27/ja/Basics/Actors/Mobility/

• you can check robot movement by using ros2 run teleop_twist_keyboard teleop_twist_keyboard

https://github.com/rapyuta-robotics/rclUE/assets/16463135/5a2d6126-3d5e-41eb-9e51-1f29d155b364

[Buddies-as-you-know]

Thank you. I did it in my simulation. Screencast from 2023年06月24日 16時53分11秒.webm

I have a question. I would like to place multiple camera actors in UE5 and bring the data to OpenCV and SLAM together with Depth.Is it possible? Do you have a recommended GIThub repository?

[yuokamoto]

• You can attach multiple cameras to robot and there is ARRROS2CameraActor which you can place as a stand alone actor. You need to change topic name and etc.
• There is a URR3DLidarComponent which publishes PointCloud2. But we haven't tested 3D lidar component much compared with 2DLidar
• Adding multiple camera may slow down simulation. There maybe more room to optimize process.

[Buddies-as-you-know]

How can I use URR3DLidarComponent?

[yuokamoto]

you can add 3D Lidar same way as camera

[Buddies-as-you-know]

Thank you. Adding multiple cameras may slow down the simulation, so I would like to get the camera footage in unreal engine 5 and display it in python, but is it possible to get it via UDP communication or something?

[yuokamoto]

not part of rclUE. you can check UE5 feature such as pixelstreaming and etc

[Buddies-as-you-know]

ok.Thank you.