Questions about evaluation.

[weimengchuan]

Hi! I have 2 questions about evaluation. I'd be grateful for you help.

1. Where to set directory of dataset before evaluation?
2. When running to the following code, the pygame window crash and the program is stuck.

   # leaderboard/team_code/lmdriver_agent.py: DisplayInterface.__init__
       self._display = pygame.display.set_mode(
           (self._width, self._height), pygame.HWSURFACE | pygame.DOUBLEBUF
       )

I tried to execute the following code separately, the pygame window continues to open.

import pygame
pygame.init()
pygame.font.init()
pygame.display.set_mode((1200, 900), pygame.HWSURFACE | pygame.DOUBLEBUF)

Thanks in advance!

[deepcs233]

Hi!

1. You can put the dataset anywhere you can access it. When you use the dataset to pretrain or finetune the model, the dataset path should be set to your dataset path.
2. Could you provide more details or the log? BTW, the two code blocks look similar? I can't understand the following context:

# leaderboard/team_code/lmdriver_agent.py: DisplayInterface.__init__

[weimengchuan]

Hi!

1. The input of the model during evaluation does not come from the dataset you provided, but from the simulator. Is that right?
2. Sorry for not describing it clearly. Here is the detailed information about the problem I encountered when evaluation. Are there anything wrong with my operating procedures?
   • My Command: First, run calar in one console.

     bash carla/CarlaUE4.sh --world-port=2000

     Second, run evaluation in another console.

     
     export PT=2000

export CARLA_ROOT=carla export CARLA_SERVER=${CARLA_ROOT}/CarlaUE4.sh export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg export PYTHONPATH=$PYTHONPATH:leaderboard export PYTHONPATH=$PYTHONPATH:leaderboard/team_code export PYTHONPATH=$PYTHONPATH:scenario_runner

export LEADERBOARD_ROOT=leaderboard export CHALLENGE_TRACK_CODENAME=SENSORS export PORT=$PT # same as the carla server port export TM_PORT=$(($PT+500)) # port for traffic manager, required when spawning multiple servers/clients export DEBUG_CHALLENGE=0 export REPETITIONS=1 # multiple evaluation runs export ROUTES=langauto/benchmark_long.xml export TEAM_AGENT=leaderboard/team_code/lmdriver_agent.py # agent export TEAM_CONFIG=leaderboard/team_code/lmdriver_config.py # model checkpoint, not required for expert export CHECKPOINT_ENDPOINT=results/lmdrive_result.json # results file

export SCENARIOS=leaderboard/data/scenarios/no_scenarios.json #town05_all_scenarios.json

export SCENARIOS=leaderboard/data/official/all_towns_traffic_scenarios_public.json export SAVE_PATH=data/eval # path for saving episodes while evaluating export RESUME=False

echo ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py python3 -u ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py \ --scenarios=${SCENARIOS} \ --routes=${ROUTES} \ --repetitions=${REPETITIONS} \ --track=${CHALLENGE_TRACK_CODENAME} \ --checkpoint=${CHECKPOINT_ENDPOINT} \ --agent=${TEAM_AGENT} \ --agent-config=${TEAM_CONFIG} \ --debug=${DEBUG_CHALLENGE} \ --record=${RECORD_PATH} \ --resume=${RESUME} \ --port=${PORT} \ --trafficManagerPort=${TM_PORT}

- Program is stuck, and the information is shown below.

leaderboard/leaderboard/leaderboard_evaluator.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources bc1 localhost 2000 bc2 bc3 leaderboard/leaderboard/leaderboard_evaluator.py:95: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if LooseVersion(dist.version) < LooseVersion('0.9.10'): bc4 /root/anaconda3/envs/lmdrive/lib/python3.8/site-packages/diffusers/models/cross_attention.py:30: FutureWarning: Importing from cross_attention is deprecated. Please import from diffusers.models.attention_processor instead. deprecate( pygame 2.5.2 (SDL 2.28.2, Python 3.8.13) Hello from the pygame community. https://www.pygame.org/contribute.html bc5 bc

========= Preparing RouteScenario_3 (repetition 0) =========

Setting up the agent

• After checking, I find the program is stuck in the following position:

  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/leaderboard_evaluator.py(497)<module>()
  -> main()
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/leaderboard_evaluator.py(488)main()
  -> leaderboard_evaluator.run(arguments)
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/leaderboard_evaluator.py(423)run()
  -> self._load_and_run_scenario(args, config)
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/leaderboard_evaluator.py(277)_load_and_run_scenario()
  -> self.agent_instance = getattr(self.module_agent, agent_class_name)(args.agent_config)
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/autoagents/autonomous_agent.py(45)__init__()
  -> self.setup(path_to_conf_file)
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/team_code/lmdriver_agent.py(176)setup()
  -> self._hic = DisplayInterface()
  > /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/team_code/lmdriver_agent.py(81)__init__()
  -> self._display = pygame.display.set_mode(

  I find that when executing this code self._display = pygame.display.set_mode, a pygame window open but immediately collapsed. Afterwards, the program will remain stuck here.

[deepcs233]

Hi!

1. Yes. Our pipeline is based on the closed-loop and end-to-end setting. The evaluation needs to interact with the simulator in a real-time manner.
2. Sorry, I can't reproduce the error and I haven't encountered this problem. Did you run it on a docker/server environment? Maybe you can try a newer version of pygame. Or remove pygame.HWSURFACE | pygame.DOUBLEBUF when you initialize the display. Or comment on corresponding pygame codes to run the evaluation without the GUI.

[weimengchuan]

Thanks for your fast response. I will try according to your recommend.