Leaderboard 2.0

[abol-karimi]

Is the code compatible with Leaderboard 2.0? The documentation refers to Carla 0.9.10.1, but I see that your team has a Leaderboard 2.0 submission as well.

I'm looking for state-of-the-art AV agents for Carla 0.9.14 to use in our ongoing work on scenario-based testing of AV stacks. It would be great if I can use TF++. Thank you!

[Kait0]

The codebase here is not compatible with the leaderboard 2.0/Carla 0.9.14 out-of-the-box, but its not too hard to adapt it. You will need to keep the team_code folder and replace the leaderboard 1.0 folders with the respective 2.0 variant. Also you might need to change some minor things for compatibility. We did this for the leaderboard 2.0 submission you mentioned. I think the model inference code was not a big problem. The data collection code is a bit trickier, the rendering code for the BEV semantic labels runs into some problems in Town 12 and 13 since they are too large for the rendering libraries. Also sensor replacement (which we used for shift augmentation during data collection), seems to not work in the large town 12 and 13 (works in all the small towns so likely a simulator bug). If you only work with the old towns the different simulator version shouldn't be much of a problem.

The leaderboad 2.0 submission you mentioned is the TF++ WP ensemble from this repository, with the code slightly adapted to work with the leaderboard 2.0 infrastructure, but with otherwise no changes. We called it zero-shot because the method was not re-trained. So it does not know the new scenarios, towns or vehicle physics. That model is the best publicly available CARLA model right now, so we wanted to see how well it works out-of-the-box.

Retraining end-to-end methods for leaderboard 2.0 is tricky right now because there is no good dataset yet, the challenge winner also mentioned that. That's why the winning methods were modular pipelines this time.

For leaderboard 1.0 everybody collected datasets using privileged planners to automatically collect data. The best privileged autopilot (the one from this repo) only gets around ~2 DS on leaderboard 2.0, upper bounding end-to-end methods. So leaderboard 2.0 for now is first a privileged planning challenge, since human data collection is often too expensive for academic teams.

[abol-karimi]

Thanks a lot! I'll try to adapt it to Carla 0.9.14 to just use the pretrained model in a small town e.g. Town05, and will report back how it goes.

[abol-karimi]

I replaced the leaderboard and scenario_runner folders with the Leaderboard-2.0 branches of the corresponding repos.

However, I can't run the evaluation, due to the following error:

user@machine:~/carla_garage_fork$ leaderboard/leaderboard/leaderboard_evaluator.py --agent-config pretrained_models/longest6/tfpp_all_0 --agent /home/carla/carla_garage_fork/team_code/sensor_agent.py --routes leaderboard/data/only_one_town.xml --checkpoint pretrained_models/longest6/tfpp_all_0/results.json
leaderboard/leaderboard/leaderboard_evaluator.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API 
WARNING: Client API version     = 64ec6e11 
WARNING: Simulator API version  = 0.9.14 
leaderboard/leaderboard/leaderboard_evaluator.py:83: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(dist.version) < LooseVersion('0.9.10'):
Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 476, in <module>
    main()
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 465, in main
    leaderboard_evaluator = LeaderboardEvaluator(arguments, statistics_manager)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 89, in __init__
    self.module_agent = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/carla/carla_garage_fork/team_code/sensor_agent.py", line 21, in <module>
    from config import GlobalConfig
ImportError: cannot import name 'GlobalConfig' from 'config' (unknown location)

even though ~/carla_garage_fork/team_code is in my PYTHONPATH.

[Kait0]

is there another file now that is called config? Perhaps need to rename it.

[abol-karimi]

Ah, yes! Carla has a config module as well. If I rename team_code/config.py then pickle cannot load the config file: https://github.com/autonomousvision/carla_garage/blob/fb4b180332e465c53095ad6de29429442664882c/team_code/sensor_agent.py#L66

I was able to fix the issue by

1. removing the team_code from PYTHONPATH,
2. replacing from config import with from team_code.config import,
3. converting the config.pickle files in the pretrained models with the following code:

   
   import pickle
   import jsonpickle
   from pathlib import Path

for p in Path('.').glob('**/config.pickle'): with open(p, 'rb') as f: config_obj = pickle.load(f) with open(p.parent/'config.json', 'w') as f: f.write(jsonpickle.encode(config_obj))

4. replacing `config.GlobalConfig` with `team_code.config.GlobalConfig`, and finally
5. using `jsonpickle` (instead of `pickle`) to load the configs:

with open(os.path.join(path_to_conf_file, 'config.json'), 'r') as args_file:
  loaded_config = jsonpickle.decode(args_file.read())

[Kait0]

thanks for reporting your solution. Didn't know about jsonpickle sounds useful. Pickle always had the disadvantage that its not human readable.

[orangelee89]

@abol-karimi
hi， I saw that you've posted carla garage in your github , Do the repo adept leaderboard v2? If it does,which version of carla did you use ? 0.9.14?

[Kait0]

perhaps a small update on this issue. We have released a expert to collect data for CARLA leaderboard v2 now. It is available on the leaderboard_2 branch. We also plan to release training code for TransFuser++ for leaderboard 2.0 at a later time.