Segmentation fault during evaluation

[Treemann]

Hi, I could train the model, but met segmentation fault during evaluation. Did anyone meet similar problem?

CUDA_VISIBLE_DEVICES=0 leaderboard/scripts/run_evaluation.sh 4.24.3-0+++UE4+Release-4.24 518 0 Disabling core dumps. leaderboard/leaderboard/leaderboard_evaluator.py leaderboard/scripts/run_evaluation.sh: line 44: 646601 Segmentation fault (core dumped) python3 -u ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py --scenarios=${SCENARIOS} --routes=${ROUTES} --repetitions=${REPETITIONS} --track=${CHALLENGE_TRACK_CODENAME} --checkpoint=${CHECKPOINT_ENDPOINT} --agent=${TEAM_AGENT} --agent-config=${TEAM_CONFIG} --debug=${DEBUG_CHALLENGE} --record=${RECORD_PATH} --resume=${RESUME} --port=${PORT} --trafficManagerPort=${TM_PORT}

[deepcs233]

Hi!

It's a common issue of the Carla simulator. You must restart the simulator and re-run the evaluation. It is advisable to restart the simulator each time you run an evaluation.

[Treemann]

@deepcs233 Thanks for your reply! Restarting Carla seemed not work for my issue...I notice that your script leaderboard/scripts/run_evaluation.sh include starting Carla and calling the evaluation code. I tried running Carla and the evaluation code in two windows separately, Carla ran well but the issue still occurred when running the evaluation code.

[deepcs233]

Can you check your Carla API version? The mismatch also causes this problem.

[fengrussell]

Maybe you are running the wrong version of python, please confirm your python path through which python

[Treemann]

$ pip list|grep carla carla 0.9.15

I try to install carla==0.9.10.1 according to your instruction in README but it could not find the exact version.

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com ERROR: Could not find a version that satisfies the requirement carla==0.9.10.1 (from versions: 0.9.5, 0.9.15) ERROR: No matching distribution found for carla==0.9.10.1

I think it's related to my python version (3.9). I notice that you mention the python version in README (conda create -n lmdrive python=3.8) and the Carla package seems related to python3.7 (./carla/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg), should I create a new env with python3.7 or python3.8?

[Treemann]

Maybe you are running the wrong version of python, please confirm your python path through which python

Agree. I'll try with a new version of python. Which version do you use, 3.7 or 3.8? @fengrussell @deepcs233

[deepcs233]

I prefer 3.8. But if you only need to run the evaluation, 3.7 is also ok.

[fengrussell]

Maybe you are running the wrong version of python, please confirm your python path through which python

Agree. I'll try with a new version of python. Which version do you use, 3.7 or 3.8? @fengrussell @deepcs233

Sorry, I didn't explain clearly. Please first confirm whether which python is the correct version in your conda environment.

[Treemann]

I tried creating env with python3.8, but there is no carla with version of 0.9.10.1 in the pip source, so I tried with the nearest version 0.9.12. It seemed the evaluation program could go further but finally segmentation fault occurred again. I wonder which version of carla do you install with pip and any suggestion to solve the new issue? Thanks!

$ CUDA_VISIBLE_DEVICES=0 leaderboard/scripts/run_evaluation.sh
4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.
leaderboard/leaderboard/leaderboard_evaluator.py
leaderboard/leaderboard/leaderboard_evaluator.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources
localhost 16672
leaderboard/leaderboard/leaderboard_evaluator.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if LooseVersion(dist.version) < LooseVersion('0.9.10'): /group/ossmodelzoo/rongzhangzheng/env/miniconda3/envs/lmdrive-py38/lib/python3.8/site-packages/diffusers/models/cross_attention.py:30: FutureWarning: Importing from cross_attention is deprecated. Please import from diffusers.models.attention_processor instead. deprecate( pygame 2.5.2 (SDL 2.28.2, Python 3.8.18) Hello from the pygame community. https://www.pygame.org/contribute.html

========= Preparing RouteScenario_3 (repetition 0) =========

Setting up the agent Fatal Python error: Segmentation fault

[fengrussell]

I tried creating env with python3.8, but there is no carla with version of 0.9.10.1 in the pip source, so I tried with the nearest version 0.9.12. It seemed the evaluation program could go further but finally segmentation fault occurred again. I wonder which version of carla do you install with pip and any suggestion to solve the new issue? Thanks!

$ CUDA_VISIBLE_DEVICES=0 leaderboard/scripts/run_evaluation.sh 4.24.3-0+++UE4+Release-4.24 518 0 Disabling core dumps. leaderboard/leaderboard/leaderboard_evaluator.py leaderboard/leaderboard/leaderboard_evaluator.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources localhost 16672 leaderboard/leaderboard/leaderboard_evaluator.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if LooseVersion(dist.version) < LooseVersion('0.9.10'): /group/ossmodelzoo/rongzhangzheng/env/miniconda3/envs/lmdrive-py38/lib/python3.8/site-packages/diffusers/models/cross_attention.py:30: FutureWarning: Importing from cross_attention is deprecated. Please import from diffusers.models.attention_processor instead. deprecate( pygame 2.5.2 (SDL 2.28.2, Python 3.8.18) Hello from the pygame community. https://www.pygame.org/contribute.html ========= Preparing RouteScenario_3 (repetition 0) =========

Setting up the agent Fatal Python error: Segmentation fault

It seems to be an exception in pygame. According to my experience, some terminals may cause exceptions. It is recommended to try another PC remotely.

[deepcs233]

@fengrussell Thank you for your kind help!

@Treemann Hi! 0.9.10.1 is the version of the Carla server. This version number has nothing to do with the Carla API. I just checked my Carla API version in my py3.8 conda environment (ubuntu) and it's 0.9.15 . Please refer to the following command:

pip uninstall carla
pip install carla

[Treemann]

Thanks for your reply! @deepcs233 Yeah, I installed the same carla version (0.9.15) at first, and the error seemed not related to the version. I traced the code and found the error was by the code in leaderboard/team_code/lmdriver_agent.py, which was mentioned by @fengrussell . I'll check this issue of pygame. (FYI: I used "remote desktop connection" to connect the remote Ubuntu server)

class DisplayInterface(object):
    def __init__(self):
    ......
        self._display = pygame.display.set_mode(
            (self._width, self._height), pygame.HWSURFACE | pygame.DOUBLEBUF
        )
    ......

[dingli-dean]

In https://github.com/opendilab/LMDrive/issues/14#issuecomment-1898154122, the author change the PythonAPI version into 0.9.12 and it works fine. However, when I use simulator and PythonAPI with different versions, Segmentation fault occurs. Does anyone face the same situation?

The port is openning for sure.

Python 3.8.18 | packaged by conda-forge | (default, Dec 23 2023, 17:21:28) Type 'copyright', 'credits' or 'license' for more information IPython 8.12.3 -- An enhanced Interactive Python. Type '?' for help. In [1]: import carla In [2]: client = carla.Client('localhost', 6008) In [3]: traffic_manager = client.get_trafficmanager(6009) WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API WARNING: Client API version = 0.9.12 WARNING: Simulator API version = 784d9b9f Segmentation fault

[pastapeter]

@Treemann
Hi I'm facing the same issue, and also i'm using VNC with remote Ubuntu server. Do you have solution for this issue?

[pastapeter]

I just deleted the self._display in lmdrive_agent.py this issue has been solved