search

carla-simulator / carla

Open-source simulator for autonomous driving research.
http://carla.org
MIT License
11.16k stars 3.6k forks source link

The main server can't receive sensor data from the subordinate servers in a multi-GPU setup. #7994

Open wanqiangzhou opened 2 months ago

wanqiangzhou commented 2 months ago

CARLA version: CARLA_0.9.15 Platform/OS: Ubuntu 20.04 Problem you have experienced: Two computers are configured for multi-GPU operation. The main server is unable to receive sensor data from the subordinate servers. What you expected to happen: Two computers successfully running in a multi-GPU setup. Steps to reproduce: Step 1: PC1 and PC2 are directly connected via an Ethernet cable. PC1 has IP address 192.168.100.1, and PC2 has IP address 192.168.100.2. Step 2: PC1 starts the Carla main server using the following command. ./CarlaUE4.sh -nullrhi Step 3: PC2 starts the Carla main server using the following command. ./CarlaUE4.sh -RenderOffScreen -carla-rpc-port=3000 -carla-primary-host=192.168.100.1 -ini [/Script/Engine.RendererSettings]:r.GraphicsAdapter=0 Step 4: PC1 started the Carla Python client program. python3 manual_control.py --sync The result: pygame shows a black screen. I suspect it's because the main server on one of the PCs cannot fetch video data from the server, causing the Carla client to fail in receiving sensor data from the Carla server. 20240730-112806

I ultimately want to achieve multi-GPU functionality across multiple computers. Now, I'm facing the issue mentioned above. Can anyone help me with this?

veinlove55 commented 2 months ago

Did anyone run the multi-gpu feature in different PCs successfully?

veinlove55 commented 1 month ago

I'm trying start carla server in 2 PCs with each 2 GPUs. 1st PC ip is 192.168.100.1, 2nd PC ip is 192.168.100.2 In the 1st PC, I start 3 command

  1. ./CarlaUE4.sh -nullrhi
  2. ./CarlaUE4.sh -RenderOffScreen -carla-rpc-port=3000 -carla-primary-host=127.0.0.1 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=0
  3. ./CarlaUE4.sh -RenderOffScreen -carla-rpc-port=4000 -carla-primary-host=127.0.0.1 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=1 In the 2nd PC, I start 2 command
  4. ./CarlaUE4.sh -RenderOffScreen -carla-rpc-port=5000 -carla-primary-host=192.168.100.1 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=0
  5. ./CarlaUE4.sh -RenderOffScreen -carla-rpc-port=6000 -carla-primary-host=192.168.100.1 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=1 Setting carla log level to debug, I got this log. The simulator notice 2nd PC's port correctly, however the IP is wrong, still using localhost. DEBUG: streaming client: connecting to 127.0.0.1:3001 DEBUG: streaming client: connected to 127.0.0.1:3001 DEBUG: streaming client: sending stream id 2 DEBUG: calling sensor Listen() Actor 56 (sensor.camera.rgb)DEBUG: streaming client: connecting to DEBUG: Actor 56 (sensor.camera.rgb) : subscribing to stream 127.0.0.1:4001 DEBUG: streaming client: connected to 127.0.0.1:4001 DEBUG: streaming client: sending stream id 58 DEBUG: calling sensor Listen() Actor 50 (sensor.camera.rgb) DEBUG: Actor 50 (sensor.camera.rgb) : subscribing to stream DEBUG: streaming client: connecting to 127.0.0.1:5001 INFO: streaming client: connection failed: Connection refused DEBUG: calling sensor Listen() Actor 51 (sensor.camera.rgb) DEBUG: Actor 51 (sensor.camera.rgb) : subscribing to stream DEBUG: streaming client: connecting to 127.0.0.1:6001 INFO: streaming client: connection failed: Connection refused
veinlove55 commented 1 month ago

Hello @bernatx , could you take a glance at this issue, any suggestion?

Blyron commented 1 month ago

We have never tried that configuration. It should be possible but we haven't tried. So we cannot give any hints

Blyron commented 1 month ago

And please, do not tag community developers directly if they are not in the conversation.

veinlove55 commented 1 month ago

We have never tried that configuration. It should be possible but we haven't tried. So we cannot give any hints

Sorry for being so presumptuous and directly tagging the developer, it will never happen again. My requirement is multi-camera simulation(4K * 7 camera, 11 camera), but I found less information about this topic. Thus I'm trying to find a solution for distributed GPU solution. From the debug info, "DEBUG: streaming client: connecting to 127.0.0.1:5001 ", I think the secondary server's rpc port(5001) is received correctly, but there's no interface for the IP. I write the IP hard code in this file "LibCarla/source/carla/streaming/detail/tcp/Client.cpp", luckily, it works.

So, could you consider adding an interface for secondary IP?