search

huawei-noah / SMARTS

Scalable Multi-Agent RL Training School for Autonomous Driving
MIT License
957 stars 190 forks source link

Help running 2022 NeurIPS Competition agents #1933

Closed atanas-kom closed 1 year ago

atanas-kom commented 1 year ago

High Level Description

Hello, I am trying to run a benchmark on the 2022 NeurIPS Competition agents which were added to SMARTS. I followed all the instructions from the ReadTheDocs page and executed the following steps:

cd <path>/SMARTS scl zoo install zoo/policies/interaction_aware_motion_prediction scl benchmark run driving_smarts==0.0 zoo.policies:interaction-aware-motion-prediction-agent-v0 --auto-install

The installation comletes successfully but when runnung the benchmark I get the following error:

(.venv) atanas@virtualbox:~/SMARTS$ scl benchmark run driving_smarts==0.0 zoo.policies:interaction-aware-motion-prediction-agent-v0 --auto-install
Requirement already satisfied: ray<=2.2.0,>2.0 in ./.venv/lib/python3.8/site-packages (2.2.0)
Requirement already satisfied: numpy>=1.16 in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (1.23.1)
Requirement already satisfied: click>=7.0 in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (8.1.3)
Requirement already satisfied: aiosignal in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (1.3.1)
Requirement already satisfied: grpcio>=1.32.0 in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (1.53.0)
Requirement already satisfied: jsonschema in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (4.17.3)
Requirement already satisfied: requests in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (2.28.2)
Requirement already satisfied: frozenlist in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (1.3.3)
Requirement already satisfied: virtualenv>=20.0.24 in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (20.21.0)
Requirement already satisfied: attrs in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (22.2.0)
Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (3.20.1)
Requirement already satisfied: filelock in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (3.10.7)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (1.0.5)
Requirement already satisfied: pyyaml in ./.venv/lib/python3.8/site-packages (from ray<=2.2.0,>2.0) (6.0)
Requirement already satisfied: platformdirs<4,>=2.4 in ./.venv/lib/python3.8/site-packages (from virtualenv>=20.0.24->ray<=2.2.0,>2.0) (3.2.0)
Requirement already satisfied: distlib<1,>=0.3.6 in ./.venv/lib/python3.8/site-packages (from virtualenv>=20.0.24->ray<=2.2.0,>2.0) (0.3.6)
Requirement already satisfied: importlib-resources>=1.4.0 in ./.venv/lib/python3.8/site-packages (from jsonschema->ray<=2.2.0,>2.0) (5.12.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in ./.venv/lib/python3.8/site-packages (from jsonschema->ray<=2.2.0,>2.0) (0.19.3)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in ./.venv/lib/python3.8/site-packages (from jsonschema->ray<=2.2.0,>2.0) (1.3.10)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.8/site-packages (from requests->ray<=2.2.0,>2.0) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.8/site-packages (from requests->ray<=2.2.0,>2.0) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.venv/lib/python3.8/site-packages (from requests->ray<=2.2.0,>2.0) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.8/site-packages (from requests->ray<=2.2.0,>2.0) (2022.12.7)
Requirement already satisfied: zipp>=3.1.0 in ./.venv/lib/python3.8/site-packages (from importlib-resources>=1.4.0->jsonschema->ray<=2.2.0,>2.0) (3.15.0)
Traceback (most recent call last):
  File "/home/atanas/SMARTS/.venv/bin/scl", line 8, in <module>
    sys.exit(scl())
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/cli/benchmark.py", line 79, in run
    run_benchmark(
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/smarts/benchmark/__init__.py", line 109, in run_benchmark
    entrypoint(
  File "/home/atanas/SMARTS/.venv/lib/python3.8/site-packages/smarts/benchmark/entrypoints/benchmark_runner_v0.py", line 204, in benchmark_from_configs
    assert agent_args, f"""
AssertionError: 
    Cannot resolve `zoo.policies:interaction-aware-motion-prediction-agent-v0`. This should be in a location that can be resolved by python
    in python's `sys.path`.  (e.g. `<path>/custom/__init__.py` or`<path>/custom.py`) 

    Please ensure agent configuration:
      - file exists
      - file path is correct
      - contains correct data

    The benchmark cannot continue.

How can I resolve this? Thank you

Version

SMARTS 1.0.9 (latest)

Operating System

Ubuntu 20.04

Problems

No response

Adaickalavan commented 1 year ago

Hi @atanas-kom,

  1. Open a new Terminal A and setup SMARTS

    # In Terminal A
$ cd /path/SMARTS
# Delete any pre-existing virtual environments
$ python3.8 -m venv ./.venv
$ source ./.venv/bin/activate
$ pip install --upgrade pip wheel
$ pip install -e .[camera_obs]

  2. To visualize in Envision

    • Go to the file: SMARTS/smarts/benchmark/driving_smarts/v0/config.yaml and make the following change. 
      benchmark.shared_env_kwargs.headless: False
    • Open a new Terminal B and execute the following: 
      # In Terminal B
$ cd /path/SMARTS
$ source ./.venv/bin/activate
$ scl envision start
# Open http://localhost:8081/

  3. Run a trained agent

    # In Terminal A
$ cd /path/SMARTS
$ scl zoo install <agent path>
$ scl benchmark run driving_smarts==0.0 <agent_locator> --auto_install

  4. Zoo agents

    • For the time being, please try only the following zoo agents by replacing the <agent_path> and <agent_locator> appropriately in the commands.
      • control-and-supervised-learning-agent-v0
      • agent_locator = zoo.policies:control-and-supervised-learning-agent-v0
      • agent_path = zoo/policies/control_and_supervised_learning
      • discrete-soft-actor-critic-agent-v0
      • agent_locator = zoo.policies:discrete-soft-actor-critic-agent-v0
      • agent_path = zoo/policies/discrete_soft_actor_critic
    • During development, both the SMARTS repository and the smarts-project.rl repository were being updated at the same time resulting in a package dependency mismatch for interaction-aware-motion-prediction-agent-v0. This will be rectified at the earliest by pushing a new package for interaction-aware-motion-prediction-agent-v0 to pypi. Until then, kindly continue experimenting with the other zoo agents.