search

bradyz / 2020_CARLA_challenge

"Learning by Cheating" (CoRL 2019) submission for the 2020 CARLA Challenge
181 stars 49 forks source link

Updated image model weights for CARLA 0.9.10 #33

Closed aaronh65 closed 3 years ago

aaronh65 commented 3 years ago

Hi Brady,

As the title says - is the LBC team planning on releasing weights for an image model trained on CARLA 0.9.10? I tried deploying the weights (linked in the README) on the Leaderboard routes in 0.9.10, but nearly every route had a 0% RouteCompletion rate. I suspect it's because those weights were trained in CARLA 0.9.9, which looks visually dissimilar to 0.9.9

bradyz commented 3 years ago

hm this seems a bit odd - the weights that are in the README score a 8.94 on the carla challenge (0.9.10.1) leaderboard link

some things i've noticed about the model is that it does pretty poorly on towns 1-2, but can get a nonzero score on the other towns

aaronh65 commented 3 years ago

Yeah I've been trying to wrap my head around this too - there's a pretty big disconnect between the 0.9.10.1 Leaderboard scores on the website and the performance I'm seeing on my end. As far as I know, the repository seems very plug-and-play: download the repository, install CARLA 0.9.10.1, download the weights and run the image agent. Regardless, I'm still seeing 0% on nearly every route given in leaderboard/data/routes_* no matter which town is being used.

I was wondering if it might be due to something like package version mismatch or floating point precision, but my pytorch/pytorch-lightning/torch/torchvision versions line up with what's specified in carla_project/requirements.txt.

bradyz commented 3 years ago

have you visualized the model's predictions via the HAS_DISPLAY env variable? I'll also try to investigate this over the weekend!

aaronh65 commented 3 years ago

Thanks for the fast responses! I've visualized them, and I'm trying to figure out how to interpret the images. It looks like the brake is almost always set to True which seems to be the behavior across all of the routes that causes a near 0% RouteCompletion. The below DEBUG images are from deploying image_agent on routes_devtest/route_01 and routes_devtest/route_02, which are respectively deployed in Town3 and Town4.

lbc_stopped_devtest_01 lbc_stopped_devtest_02

bradyz commented 3 years ago

can you make sure to disable these black squares? i forget the flag for this but i believe it is DEBUG_CHALLENGE = 0

aaronh65 commented 3 years ago

Setting DEBUG_CHALLENGE=0 removed the black squares and seems to have resolved the perpetual braking issue, with the agent hitting a 17% RouteCompletion rate on the route I was running (at which point I stopped the run)

After thinking it through, I guess those black squares were present in the image that's passed to the model (I had thought they were only visible to the CARLA spectator or were drawn post model-inference). Seems like an obvious observation in retrospect...