search

bradyz / 2020_CARLA_challenge

"Learning by Cheating" (CoRL 2019) submission for the 2020 CARLA Challenge
181 stars 49 forks source link

Should I expect the LBC agent code to run properly on CARLA 0.9.10.1? #21

Closed aaronh65 closed 3 years ago

aaronh65 commented 3 years ago

The CARLA AD Leaderboard website has updated their "Getting started" docs to reflect an upgrade to CARLA 0.9.10.1. I tried deploying a pre-trained LBC model (downloading the weights recommended in this repo's README) in the new Leaderboard binary release, but the agent rarely moved in the routes I tested it in.

Should I be expecting the agent to move faster? I can see reasonable waypoints being produced in the image view, but the brake is almost always set to true, even if it's at a traffic signal that is green.

bradyz commented 3 years ago

i havent ported this to use 0.9.10.1 yet, but will be working on this in the coming week

aaronh65 commented 3 years ago

Hey Brady, thanks for the response.

I looked on the Leaderboard website recently and saw that LBC has results up for version 0.9.10.1 - are there plans to make that code publicly available?

bradyz commented 3 years ago

should be ready now! try the newest commit

aaronh65 commented 3 years ago

Thanks Brady! Will the team be providing updated weights for any of the networks?

bradyz commented 3 years ago

i won't be retraining this model using data collected in 0.9.10.1,
but our old model weights are public and you can try those!

aaronh65 commented 3 years ago

Just tried them out and ran the image agent on every route available in the data directory (mixed with the all scenarios json file) - there were only 4 routes where the RouteCompletion metric was not zero.

The results on the Leaderboard website indicate a 17% completion rate (I assume on some held-out test set), which is far higher than what I saw on the training + testing routes. Is this known behavior, or am I doing something wrong?