can not evaluate auto pilot or transfuser

[shenzo-ai]

i'm following the steps described here , yet im getting this when i try to evaluate , would you please help me understand ?

/leaderboard/leaderboard/leaderboard_evaluator_local.py:89: DeprecationWarning: distutils Ver sion classes are deprecated. Use packaging.version instead.
if LooseVersion(dist.version) < LooseVersion('0.9.10'):

Registering the global statistics

and then it exit execution

[Kait0]

Sound like the same issue as here. You probably already completed all the routes once. Try setting RESUME=0 Perhaps your autopilot also crashed for another reason earlier.

[shenzo-ai]

i changed resume =0 but im getting the same output

[shenzo-ai]

a friend of mine last time solved it by changing something related to checkpoints , does this make sense to you ?

[Kait0]

Registering the global statistics comes when it is done with all the routes. What is the route file you are using?

[shenzo-ai]

export ROUTES=${WORK_DIR}/leaderboard/data/longest6/longest6.xml im using this one

[Kait0]

What is the content of the result .json file? Is is set via the --checkpoint option of the leaderboard_evaluator_local.py

[shenzo-ai]

for the content of the result .json file , it is as follows :

"_checkpoint": {
    "global_record": {
        "index": -1,
        "infractions": {
            "collisions_layout": 0.0,
            "collisions_pedestrian": 0.0,
            "collisions_vehicle": 0.0,
            "outside_route_lanes": 0.0,
            "red_light": 0.0,
            "route_dev": 0.0,
            "route_timeout": 0.0,
            "stop_infraction": 0.0,
            "vehicle_blocked": 0.0
        },
        "meta": {
            "exceptions": [
                [
                    "RouteScenario_0",
                    0,
                    "Failed - Agent couldn't be set up"
                ],
                [
                    "RouteScenario_1",
                    1,
                    "Failed - Agent couldn't be set up"
                ],
                [
                    "RouteScenario_2",
                    2,
                    "Failed - Agent couldn't be set up"
                ],
                [
                    "RouteScenario_3",
                    3,
                    "Failed - Agent couldn't be set up"
                ],
                [
                    "RouteScenario_4",
                    4,
                    "Failed - Agent couldn't be set up"
                ],
                [
                    "RouteScenario_5",
                    5,
                    "Failed - Agent couldn't be set up"

i dont understand the second part of your answer brother

[Kait0]

"Failed - Agent couldn't be set up" This means there was some crash when your agent ran the route the first time you tried. Can you try to delete the file, and rerun the autopilot ( resume =0 should do this automatically but maybe there was another problem there). You should get a more meaningful error message then.

[shenzo-ai]

this time i'm getting the following error :

                                                                                         │

========= Preparing RouteScenario_0 (repetition 0) ========= │

Setting up the agent │ │ The sensor's configuration used is invalid: │ You are submitting to the wrong track [Track.SENSORS]! │ │ Traceback (most recent call last): │ File "./leaderboard/leaderboard/leaderboard_evaluator_local.py", line 279, in _load_andrun│ scenario │ AgentWrapper.validate_sensor_configuration(self.sensors, track, args.track) │ File "/home/shenzo/transfuser/leaderboard/leaderboard/autoagents/agent_wrapper_local.py", li│ ne 203, in validate_sensor_configuration │ raise SensorConfigurationInvalid("You are submitting to the wrong track [{}]!".format(Trac│ k(selected_track))) │ leaderboard.envs.sensor_interface.SensorConfigurationInvalid: You are submitting to the wrong │ track [Track.SENSORS]! │ Registering the route statistics

[Kait0]

The leaderboard_evaluator_local.py you are running has a command line option --track. For TransFuser it needs to be set to --track=SENSORS For the autopilot it needs to be set to --track=MAP The code is complaining that you set the wrong option. Try changing it to --track=MAP

[shenzo-ai]

i changed my run_eval file to the following : xport CARLA_ROOT=${1:-/home/shenzo/transfuser/carla} export WORK_DIR=${2:-/home/shenzo/transfuser}

export CARLA_SERVER=${CARLA_ROOT}/CarlaUE4.sh export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg export SCENARIO_RUNNER_ROOT=${WORK_DIR}/scenario_runner export LEADERBOARD_ROOT=${WORK_DIR}/leaderboard export PYTHONPATH="${CARLA_ROOT}/PythonAPI/carla/":"${SCENARIO_RUNNER_ROOT}":"${LEADERBOARD_ROOT}":${PYTHONPATH}

export SCENARIOS=${WORK_DIR}/leaderboard/data/maps/Town01_details.json export ROUTES=${WORK_DIR}/leaderboard/data/longest6/longest6.xml export REPETITIONS=1 export CHALLENGE_TRACK_CODENAME=maps export CHECKPOINT_ENDPOINT=${WORK_DIR}/results/transfuser_longest6.json export TEAM_AGENT=${WORK_DIR}/team_code_autopilot/autopilot.py export TEAM_CONFIG=${WORK_DIR}/model_ckpt/transfuser/args.txt export DEBUG_CHALLENGE=0 export RESUME=0 export DATAGEN=0

python3 ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator_local.py \ --scenarios=${SCENARIOS} \ --routes=${ROUTES} \ --repetitions=${REPETITIONS} \ --track=${CHALLENGE_TRACK_CODENAME} \ --checkpoint=${CHECKPOINT_ENDPOINT} \ --agent=${TEAM_AGENT} \ --agent-config=${TEAM_CONFIG} \ --debug=${DEBUG_CHALLENGE} \ --resume=${RESUME}

is there something i can change ? (thank you immensely for your time )

[Kait0]

The line export CHALLENGE_TRACK_CODENAME=maps needs to be CHALLENGE_TRACK_CODENAME=MAP without the s and capital.

the line export SCENARIOS=${WORK_DIR}/leaderboard/data/maps/Town01_details.json should be export SCENARIOS=${WORK_DIR}/leaderboard/data/longest6/eval_scenarios.json if you want to evaluate longest6. (Town01_details.json is some visualization file)

the line export TEAM_CONFIG=${WORK_DIR}/model_ckpt/transfuser/args.txt should be export TEAM_CONFIG=${WORK_DIR}/model_ckpt/transfuser (though it is only relevant for TransFuser not the autopilot.)

[shenzo-ai]

by changing to what you suggested autopilot run smoothly , would please share the correct run_eval file to evaluate transfuser or should i just copy the one here in the repo ?

[shenzo-ai]

i did use the run eval file published here , i got the following error :

raceback (most recent call last): │ File "./leaderboard/leaderboard/leaderboard_evaluator_local.py", line 271, in _load_and_ru│ n_scenario │ self.agent_instance = getattr(self.module_agent, agent_class_name)(args.agent_config) │ File "/home/shenzo/transfuser/leaderboard/leaderboard/autoagents/autonomous_agent.py", lin│ e 45, in init │ self.setup(path_to_conf_file) │ File "./team_code_transfuser/submission_agent.py", line 41, in setup │ args_file = open(os.path.join(path_to_conf_file, 'args.txt'), 'r') │ FileNotFoundError: [Errno 2] No such file or directory: './model_ckpt/transfuser/args.txt' │

Registering the route statistics │ │ ========= Preparing RouteScenario_35 (repetition 0) ========= │ Setting up the agent │ │ Could not set up the required agent: │ [Errno 2] No such file or directory: './model_ckpt/transfuser/args.txt' │ │ Traceback (most recent call last): │ File "./leaderboard/leaderboard/leaderboard_evaluator_local.py", line 271, in _load_and_ru│ n_scenario │ self.agent_instance = getattr(self.module_agent, agent_class_name)(args.agent_config) │ File "/home/shenzo/transfuser/leaderboard/leaderboard/autoagents/autonomous_agent.py", lin│ e 45, in init │ self.setup(path_to_conf_file) │ File "./team_code_transfuser/submission_agent.py", line 41, in setup │ args_file = open(os.path.join(path_to_conf_file, 'args.txt'), 'r') │ FileNotFoundError: [Errno 2] No such file or directory: './model_ckpt/transfuser/args.txt' │ Registering the route statistics │ Registering the global statistics

and it exit

[Kait0]

The one in the repository should work out of the box if you set the carla and transfuser path correctly.

Did you download and copied the model weights and args.txt file into the path/to/transfuser/team_code_transfuser/model_ckpt/transfuser/ folder?

[shenzo-ai]

yes , this is the folder that you are referring to :

tfuse) ┌──(shenzo㉿Shenzo)-[~/transfuser/model_ckpt/models_2022/transfuser] │ └─$ ls │ args.txt model_seed1_39.pth model_seed2_39.pth model_seed3_37.pth

[Kait0]

Yes but the models_2022/ subfolder is not supposed to be there. Either move the transfuser folder up by 1 folder or change the line export TEAM_CONFIG=${WORK_DIR}/model_ckpt/transfuser to export TEAM_CONFIG=${WORK_DIR}/model_ckpt/models_2022/transfuser

[shenzo-ai]

everything is working now . thank you very much brother .(let's see how can i surpass transfuser on the leaderboard within a year ).