Running the evaluation

[sqb2145]

Hi, Thank you for releasing the code and the dataset.

Sorry for the question, but I'm having trouble running the run_evaluation.sh file. I've placed the new.ckpt file and the results_TCP.json file in the outer TCP directory. And I've just edited the run_evaluation.sh line 23 to be:

export TEAM_CONFIG= new.ckpt

This results in an error which can be traced back to the watchdog.py file which says: "NoneType object has no attribute cancel"

I believe this is the case of the model file not being accessible.

Can you please tell me if this is the correct way to add the path to the model file? If not, how should I change it?

Thank You

[penghao-wu]

I think "NoneType object has no attribute cancel" is not the actual cause of the error. Could you please provide the full output?

[sqb2145]

Hi, Here is the full output:

(TCP) psu@psu-Serval:~/TCP$ sh leaderboard/scripts/run_evaluation.sh init statistics_manager begin Registering the global statistics Exception ignored in: <function LeaderboardEvaluator.del at 0x7fa81f4b5560> Traceback (most recent call last): File "leaderboard/leaderboard/leaderboard_evaluator.py", line 150, in del self._cleanup() File "leaderboard/leaderboard/leaderboard_evaluator.py", line 183, in _cleanup self._agent_watchdog.stop() File "/home/psu/TCP/scenario_runner/srunner/scenariomanager/watchdog.py", line 72, in stop self._timer.cancel() AttributeError: 'NoneType' object has no attribute 'cancel'

And this is what my run_evaluation.sh file looks like:

`#!/bin/bash

export CARLA_ROOT= PATH_TO_CARLA

export CARLA_SERVER=${CARLA_ROOT}/CarlaUE4.sh

export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI

export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla

export PYTHONPATH=$PYTHONPATH:/home/psu/carla/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg export PYTHONPATH=$PYTHONPATH:leaderboard export PYTHONPATH=$PYTHONPATH:leaderboard/team_code export PYTHONPATH=$PYTHONPATH:scenario_runner

export LEADERBOARD_ROOT=leaderboard export CHALLENGE_TRACK_CODENAME=SENSORS export PORT=2000 export TM_PORT=8000 export DEBUG_CHALLENGE=0 export REPETITIONS=3 # multiple evaluation runs export RESUME=True

TCP evaluation

export ROUTES=/home/psu/TCP/leaderboard/data/evaluation_routes/routes_lav_valid.xml export TEAM_AGENT=leaderboard/team_code/tcp_agent.py export TEAM_CONFIG=new.ckpt export CHECKPOINT_ENDPOINT=results_TCP.json export SCENARIOS=leaderboard/data/scenarios/all_towns_traffic_scenarios.json export SAVE_PATH=data/results_TCP

python3 ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py \ --scenarios=${SCENARIOS} \ --routes=${ROUTES} \ --repetitions=${REPETITIONS} \ --track=${CHALLENGE_TRACK_CODENAME} \ --checkpoint=${CHECKPOINT_ENDPOINT} \ --agent=${TEAM_AGENT} \ --agent-config=${TEAM_CONFIG} \ --debug=${DEBUG_CHALLENGE} \ --record=${RECORD_PATH} \ --resume=${RESUME} \ --port=${PORT} \ --trafficManagerPort=${TM_PORT}

` Any help is greatly appreciated. Thanks!

[lgf2023]

I have the same problem,Have you solved it yet？