Unstable Server and unable to run benchmark_agent.py

[mgmayank18]

Most of the times when I start Carla in server mode, I get the following error.

./CarlaUE4.sh -fps=10 -benchmark 4.22.1-0+++UE4+Release-4.22 517 0 Disabling core dumps. sh: xdg-user-dir: command not found lock: No locks available lock: No locks available lock: No locks available Assertion 'm->io_events_please_scan > 0' failed at pulse/mainloop.c:490, function cleanup_io_events(). Aborting. Signal 6 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=6 Malloc Size=65535 LargeMemoryPoolOffset=131119 Malloc Size=140864 LargeMemoryPoolOffset=272000 Engine crash handling finished; re-raising signal 6 for the default handler. Good bye.

It is either Signal 6 or Signal 11 during the crash.

Even if it does run, which happens rarely, upon running the next command, I get the following error.

CUDA_VISIBLE_DEVICES="1" python benchmark_agent.py --suite=town2 --model-path=ckpts/image/model-10.th --show pygame 1.9.6 Hello from the pygame community. https://www.pygame.org/contribute.html None WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API WARNING: Client API version = 0.9.6-15-g2ce2ee3b WARNING: Simulator API version = 0.9.6 0%| | 0/100 [00:00<?, ?it/s]ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_card_driver returned error: Permission denied ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_concat returned error: Permission denied ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_refer returned error: Permission denied ALSA lib conf.c:5034:(snd_config_expand) Evaluate error: Permission denied ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM default spawned 0 vehicles spawned 0 pedestrians Initial collided: False Offset: 4.0 : cannot connect to X server

Any help on this would be appreciated.

Thanks!

[bradyz]

could be related to https://github.com/carla-simulator/carla/issues/828 have you updated your nvidia drivers?

[mgmayank18]

could be related to carla-simulator/carla#828 have you updated your nvidia drivers?

I am using my lab's cluster, which has the following driver and CUDA version. Also, it is running Springdale Linux release 7.6 (Verona). The drivers seem to be more up to date than in the linked thread.

NVIDIA-SMI 418.74 Driver Version: 418.74 CUDA Version: 10.1

[bradyz]

drivers should be fine, mine read

NVIDIA-SMI 418.88 Driver Version: 418.88 CUDA Version: 10.1

I'm guessing it's the linux distro since we've tested this only on ubuntu

The next thing I'd try is to try running in a docker https://carla.readthedocs.io/en/latest/build_docker/

[dotchen]

@mgmayank18 Don't worry about the alsa errors, it is mostly likely due to your cluster does not have sound card. I also got those when benchmarking on the cluster.

For the Carla server error, if your cluster has docker or singularity installed, follow Brady's pointer above and launch inside docker image should solve it.

[mgmayank18]

@dianchen96 @bradyz. I got the server running stable by using a different node on the cluster. There might have been some issue with the one I was using before. However, I am still getting the second set of Warnings as follows. I can ignore the ALSA warnings but I'm worried about the other warnings. I am getting 0 spawned pedestrians and cars. Also, since it's a remote server, there is the issue of having no X server.

CUDA_VISIBLE_DEVICES="2" python benchmark_agent.py --suite=town2 --model-path=ckpts/image/model-10.th --show pygame 1.9.6 Hello from the pygame community. https://www.pygame.org/contribute.html None WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API WARNING: Client API version = 0.9.6-15-g2ce2ee3b WARNING: Simulator API version = 0.9.6 0%| | 0/100 [00:00<?, ?it/s]ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_card_driver returned error: Permission denied ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_concat returned error: Permission denied ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_refer returned error: Permission denied ALSA lib conf.c:5034:(snd_config_expand) Evaluate error: Permission denied ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM default spawned 0 vehicles spawned 0 pedestrians Initial collided: False Offset: 4.0 : cannot connect to X server

[mgmayank18]

Also, on running the data collection script I get the following. I'm not sure if this is the expected behaviour.

python data_collector.py --dataset_path=./dataset pygame 1.9.6 Hello from the pygame community. https://www.pygame.org/contribute.html Episode: 0%| | 0/50 [00:00<?, ?it/s]None WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API WARNING: Client API version = 0.9.6-15-g2ce2ee3b WARNING: Simulator API version = 0.9.6 A LSA lib confmisc.c:767:(parse_card) cannot find card '0'| 0/4000 [00:00<?, ?it/s] ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_card_driver returned error: Permission denied ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_concat returned error: Permission denied ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4555:(_snd_config_evaluate) function snd_func_refer returned error: Permission denied ALSA lib conf.c:5034:(snd_config_expand) Evaluate error: Permission denied ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM default spawned 100 vehicles Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position Spawn failed because of collision at spawn position spawned 250 pedestrians Teleported walker 535 to Location(x=247.813568, y=58.371265, z=0.078423) Teleported walker 664 to Location(x=390.716370, y=203.575287, z=0.214813) Teleported walker 633 to Location(x=343.534729, y=269.567078, z=0.278421) Teleported walker 908 to Location(x=127.271210, y=136.001160, z=0.278421) Teleported walker 664 to Location(x=49.925434, y=137.877991, z=0.278421) Teleported walker 918 to Location(x=280.664703, y=332.020691, z=0.221773) Teleported walker 532 to Location(x=203.924225, y=128.887848, z=0.078423) Initial collided: False : cannot connect to X server

[bradyz]

I believe you should remove the --show flag if you do not have a display, and the data_collector seems to be running fine, are you seeing frames being saved?

[mgmayank18]

I believe you should remove the --show flag if you do not have a display, and the data_collector seems to be running fine, are you seeing frames being saved?

No, the ./dataset folder is empty and the script ends after the "cannot connect to X server" message.

[dotchen]

@mgmayank18

WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API WARNING: Client API version = 0.9.6-15-g2ce2ee3b WARNING: Simulator API version = 0.9.6

This is expected and is good, because you are runining with our egg file which is compiled separately from the Carla sever.

spawned 0 vehicles spawned 0 pedestrians

This is expected, because the first test suites are the empty scenarios.

: cannot connect to X server

For data collector, try comment out this line: https://github.com/dianchen96/LearningByCheating/blob/release-0.9.6/data_collector.py#L108 and rerun it. For benchmarking, simply remove the --show flag.

Spawn failed because of collision at spawn position

This is expected. The test suite tries to randomly spawn vehicles, and if it detects a collision (spawned on top of other vehicles etc.) it respawns.

Teleported walker 535 to Location(x=247.813568, y=58.371265, z=0.078423)

This is expected. If a pedestrian is found to be stuck and possibly causing a traffic jam, we teleport the pedestrian to somewhere else.

I hope this helps and let me know if you have further questions :D