About the evaluation

[zhaoxiang0715]

Hello, sorry to bother you, I have some problems in the process of running your source code, I have these errors when I evaluate after training. --------------------TEST Agent: mmfn_imgnet.py-------------------- It will overwrite the things! ATTENTION /mnt/Data/mmfn/mmfn/data/results/only_one_town_mmfn_imgnet.json

========= Preparing RouteScenario_0 (repetition 0) =========

Setting up the agent loading model

Could not set up the required agent:

Error(s) in loading state_dict for MMFN: Missing key(s) in state_dict: "encoder.vectornet_encoder.lane_subgraph.layers.mlp_0.mlp.0.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_0.mlp.0.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_0.mlp.1.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_0.mlp.1.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_1.mlp.0.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_1.mlp.0.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_1.mlp.1.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_1.mlp.1.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_2.mlp.0.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_2.mlp.0.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_2.mlp.1.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_2.mlp.1.bias", "encoder.vectornet_encoder.pos_emb.0.weight", "encoder.vectornet_encoder.pos_emb.0.bias", "encoder.vectornet_encoder.pos_emb.1.weight", "encoder.vectornet_encoder.pos_emb.1.bias", "encoder.vectornet_encoder.pos_emb.3.weight", "encoder.vectornet_encoder.pos_emb.3.bias", "encoder.vectornet_encoder.L2L.to_qkv.weight", "encoder.vectornet_encoder.L2L.to_out.0.weight", "encoder.vectornet_encoder.L2L.to_out.0.bias", "encoder.vectornet_encoder.agent_fusion.0.weight", "encoder.vectornet_encoder.agent_fusion.0.bias", "encoder.vectornet_encoder.agent_fusion.1.weight", "encoder.vectornet_encoder.agent_fusion.1.bias", "encoder.vectornet_encoder.agent_fusion.3.weight", "encoder.vectornet_encoder.agent_fusion.3.bias", "encoder.vectornet_encoder.generator.0.weight", "encoder.vectornet_encoder.generator.0.bias", "encoder.vectornet_encoder.generator.1.weight", "encoder.vectornet_encoder.generator.1.bias", "encoder.vectornet_encoder.generator.3.weight", "encoder.vectornet_encoder.generator.3.bias".

[Kin-Zhang]

Aha, I check the scripts again, looks like the mmfn_imgnet.py previously imported MMFN from model_vec. I push again. Could you try now? https://github.com/Kin-Zhang/mmfn/blob/main/team_code/e2e_agent/mmfn_imgnet.py#L13

[zhaoxiang0715]

Aha, I check the scripts again, looks like the mmfn_imgnet.py previously imported MMFN from model_vec. I push again. Could you try now?啊哈，我再次检查脚本，看起来像之前从 model_vec 导入的 mmfn_imgnet.py MMFN。我又推了一次。你现在可以试试吗？ https://github.com/Kin-Zhang/mmfn/blob/main/team_code/e2e_agent/mmfn_imgnet.py#L13

Thank you very much. That's all. But there was a new error, and I checked the file code for the error and found that there was no definition for "smooth_queue". --------------------TEST Agent: mmfn_imgnet.py-------------------- It will overwrite the things! ATTENTION /mnt/Data/mmfn/mmfn/data/results/only_one_town_mmfn_imgnet.json

========= Preparing RouteScenario_0 (repetition 0) =========

Setting up the agent loading model load model success

Could not set up the required agent:

module 'expert_agent.common.utils' has no attribute 'smooth_queue'

[Kin-Zhang]

fix again. Could you try again and let me know if still have problem? (Looks like I forgot to delete some test functions.

[zhaoxiang0715]

fix again. Could you try again and let me know if still have problem? (Looks like I forgot to delete some test functions.再次修复。你能再试一次，如果还有问题，请告诉我吗？（看起来我忘了删除一些测试函数。

Thank you very much! The bug has been fixed, but there are still some problems.

--------------------TEST Agent: mmfn_imgnet.py-------------------- It will overwrite the things! ATTENTION /mnt/Data/mmfn/mmfn/data/results/only_one_town_mmfn_imgnet.json

========= Preparing RouteScenario_0 (repetition 0) =========

Setting up the agent loading model load model success Loading the world Running the route

Stopping the route, the agent has crashed:

forward() missing 3 required positional arguments: 'radar_adj', 'target_point', and 'velocity'

Traceback (most recent call last): File "/mnt/Data/mmfn/mmfn/leaderboard/leaderboard/scenarios/scenario_manager.py", line 152, in _tick_scenario ego_action = self._agent() File "/mnt/Data/mmfn/mmfn/leaderboard/leaderboard/autoagents/agent_wrapper.py", line 76, in call return self._agent() File "/mnt/Data/mmfn/mmfn/leaderboard/leaderboard/autoagents/autonomous_agent.py", line 115, in call control = self.run_step(input_data, timestamp) File "/home/zx/anaconda3/envs/mmfn/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context return func(*args, *kwargs) File "/mnt/Data/mmfn/mmfn/team_code/e2e_agent/mmfn_imgnet.py", line 282, in run_step target_point, gt_velocity) File "/home/zx/anaconda3/envs/mmfn/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(input, **kwargs) TypeError: forward() missing 3 required positional arguments: 'radar_adj', 'target_point', and 'velocity'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/mnt/Data/mmfn/mmfn/leaderboard/leaderboard/leaderboard_evaluator.py", line 344, in _load_and_run_scenario self.manager.run_scenario() File "/mnt/Data/mmfn/mmfn/leaderboard/leaderboard/scenarios/scenario_manager.py", line 136, in run_scenario self._tick_scenario(timestamp) File "/mnt/Data/mmfn/mmfn/leaderboard/leaderboard/scenarios/scenario_manager.py", line 159, in _tick_scenario raise AgentError(e) leaderboard.autoagents.agent_wrapper.AgentError: forward() missing 3 required positional arguments: 'radar_adj', 'target_point', and 'velocity'

Stopping the route

========= Results of RouteScenario_0 (repetition 0) ------ FAILURE =========

╒═════════════════════════════════╤═════════════════════╕ │ Start Time │ 2023-11-03 17:17:40 │ ├─────────────────────────────────┼─────────────────────┤ │ End Time │ 2023-11-03 17:17:41 │ ├─────────────────────────────────┼─────────────────────┤ │ Duration (System Time) │ 0.34s │ ├─────────────────────────────────┼─────────────────────┤ │ Duration (Game Time) │ 0.15s │ ├─────────────────────────────────┼─────────────────────┤ │ Ratio (System Time / Game Time) │ 0.441 │ ╘═════════════════════════════════╧═════════════════════╛

╒═══════════════════════╤═════════╤═════════╕ │ Criterion │ Result │ Value │ ├───────────────────────┼─────────┼─────────┤ │ RouteCompletionTest │ FAILURE │ 0.0 % │ ├───────────────────────┼─────────┼─────────┤ │ OutsideRouteLanesTest │ SUCCESS │ 0 % │ ├───────────────────────┼─────────┼─────────┤ │ CollisionTest │ SUCCESS │ 0 times │ ├───────────────────────┼─────────┼─────────┤ │ RunningRedLightTest │ SUCCESS │ 0 times │ ├───────────────────────┼─────────┼─────────┤ │ RunningStopTest │ SUCCESS │ 0 times │ ├───────────────────────┼─────────┼─────────┤ │ InRouteTest │ SUCCESS │ │ ├───────────────────────┼─────────┼─────────┤ │ AgentBlockedTest │ SUCCESS │ │ ├───────────────────────┼─────────┼─────────┤ │ Timeout │ SUCCESS │ │ ╘═══════════════════════╧═════════╧═════════╛

Registering the route statistics Registering the global statistics clean memory on no. 2428 Uncollectable garbage: [] TOTAL RUNNING TIME: --- 0.01 hours --- Exception ignored in: <function LeaderboardEvaluator.del at 0x7efecd6b5f80> Traceback (most recent call last): File "/mnt/Data/mmfn/mmfn/leaderboard/leaderboard/leaderboard_evaluator.py", line 118, in del self._cleanup() File "/mnt/Data/mmfn/mmfn/leaderboard/leaderboard/leaderboard_evaluator.py", line 130, in _cleanup if self.manager and self.manager.get_running_status() \ AttributeError: 'LeaderboardEvaluator' object has no attribute 'manager'

[Kin-Zhang]

fix again... Thanks for reminding.