search

autonomousvision / transfuser

[PAMI'23] TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving; [CVPR'21] Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
MIT License
1.04k stars 175 forks source link

Can't reproduce the same RC on longest6 Benchmark #191

Closed JinRanYAO closed 10 months ago

JinRanYAO commented 10 months ago

Hello, Thank you for your excellent job. I evaluate the model on longest6 Benchmark, with your open-sourced pretrained model and model retrained by myself, but both of them can't reproduce the same result on longest6 Benchmark, especially RC. Emseble 3 models can't get the same RC too. For example, results by result_parser(emsemble 3 open-sourced pretrained model): Avg. driving score,46.410666666666664 Avg. route completion,82.96050000000001 Avg. infraction penalty,0.591 Collisions with pedestrians,0.0660326965648726 Collisions with vehicles,1.6361434815518432 Collisions with layout,0.029347865139943377 Red lights infractions,0.1320653931297452 Stop sign infractions,0.39619617938923557 Off-road infractions,0.03546616132499308 Route deviations,0.0 Route timeouts,0.0660326965648726 Agent blocked,0.29347865139943374 I see the same question in #176 , and run CARLA Server with -opengl option, with block threshold 180.0. Compared with results in paper: image There is less collision with vehicles, but more agent blocked. Could you tell me is there something wrong? Thanks!

Kait0 commented 10 months ago

You seem to get a similar result as the mentioned issue. It doesn't look too wrong and I don't have any other particular insight here. Could you send us the information about which GPU type and operating system you used to evaluate? I have seen a couple of people have this kind of issue across various projects, so I want to keep track to see if there are any common factors.

JinRanYAO commented 10 months ago

OK, thank you for your reply. I use Ubuntu 18.04 operating system and nvidia RTX 3090 GPU.

Could you please tell me the reason if you find? Thank you very much!

Kait0 commented 10 months ago

image 30xx series GPUs are known to produce these black rendering artifacts with CARLA 0.9.10.1, that older GPUs don't produce. Made this screenshot on my local computer which also has a 3090 GPU.

Seems to be a driver bug in unreal, so I don't really have a solution. The neural network will likely be able to handle these bug if you train it with data collected with these GPUs but the pretrained checkpoints were trained with data collected by 2080ti gpus.

JinRanYAO commented 10 months ago

OK, thank you for your reply. How long does it take to re-collect 280k data with 30xx series GPUs?

Kait0 commented 10 months ago

Depends primarily how many GPUs you have. With 32x 3090 I would guess 1-2 days. If you only have a single GPU its probably too slow. You can find an example of how to parallelize data collection in our latest repo.

JinRanYAO commented 10 months ago

Oh, it is really too slow. I will try to find 20xx series GPUs to reproduce the results. Thank you very much!