Implementing Convolutinal DQN Network from the scripts

[OscarHuangWind]

Hi Edouard,

Thank you for your amazing contribution at the first place. I am currently studying DQN network(image-input with convolutional network) and want to implement it to highway-env. I have tried to run MultiLayerPercepton DQN(dqn, d2qn, d3qn), which is the default model, from the command line by following instruction and it worked.

My questions are:

1. How can I run a Convolutional DQN network with image input in highway-env environment?
2. How can I define the model(or hyper-parameters) and run Convolutional DQN network from the scripts but not from the command line ?

I will really appreciate if you reply me as soon as possible.

Oscar Huang

[eleurent]

Hi @OscarHuangWind,

The image input and convolutional DQN netwok are already implemented in highway-env and rl-agents, respecrively. Here is a script allowing you to run it (also available as a command line prompt, through configuration files):

import highway_env
from pathlib import Path
from rl_agents.agents.common.factory import load_environment, load_agent
from rl_agents.trainer.evaluation import Evaluation

env_config = {
    "id": "highway-v0",
    "lanes_count": 3,
    "vehicles_count": 15,
    "observation": {
        "type": "GrayscaleObservation",
        "observation_shape": (128, 64),
        "stack_size": 4,
        "weights": [0.2989, 0.5870, 0.1140],  # weights for RGB conversion
        "scaling": 1.75,
    },
    "policy_frequency": 2,
    "duration": 40,
}

agent_config = {
    "__class__": "<class 'rl_agents.agents.deep_q_network.pytorch.DQNAgent'>",
    "double": True,
    "loss_function": "l2",
    "optimizer": {
        "lr": 5e-4
    },
    "gamma": 0.8,
    "n_steps": 1,
    "batch_size": 32,
    "memory_capacity": 15000,
    "target_update": 50,
    "exploration": {
        "method": "EpsilonGreedy",
        "tau": 6000,
        "temperature": 1.0,
        "final_temperature": 0.05
    },
    "model": {
        "type": "ConvolutionalNetwork",
        "activation": "RELU",
        "head_mlp": {
            "type": "MultiLayerPerceptron",
            "layers": [20],
            "activation": "RELU",
            "reshape": "True"
        }
    }
}

env = load_environment(env_config)
agent = load_agent(agent_config, env)
run_directory = None
evaluation = Evaluation(env,
                        agent,
                        run_directory=Path("./logs/"),
                        num_episodes=1000,
                        sim_seed=0,
                        recover=False,
                        display_env=True,
                        display_agent=True,
                        display_rewards=True)
evaluation.train()

[PatrickYanZ]

Hi @eleurent ,

Here is the problem I run recently. Highway env seems not support DQN agent since it said compatible with discrete action space.

UserWarning: WARN: A Box observation space has an unconventional shape (neither an image, nor a 1D vector). We recommend flattening the observation to have only a 1D vector or use a custom policy to properly process the data. Actual observation shape: (5, 5)
  logger.warn(
Traceback (most recent call last):
  File "I:\Research\tcom paper\rl-agents\scripts\highway_cnn.py", line 52, in <module>
    agent = load_agent(agent_config, env)
  File "D:\Users\yanzi\miniconda3\lib\site-packages\rl_agents\agents\common\factory.py", line 42, in load_agent
    return agent_factory(env, agent_config)
  File "D:\Users\yanzi\miniconda3\lib\site-packages\rl_agents\agents\common\factory.py", line 25, in agent_factory
    agent = agent_class(environment, config)
  File "D:\Users\yanzi\miniconda3\lib\site-packages\rl_agents\agents\deep_q_network\pytorch.py", line 16, in __init__
    super(DQNAgent, self).__init__(env, config)
  File "D:\Users\yanzi\miniconda3\lib\site-packages\rl_agents\agents\deep_q_network\abstract.py", line 14, in __init__
    assert isinstance(env.action_space, spaces.Discrete) or isinstance(env.action_space, spaces.Tuple), \
AssertionError: Only compatible with Discrete action spaces.

[eleurent]

Hey, I think I solved this bug today, would you mind upgrading rl-agents and trying again?

[PatrickYanZ]

Thanks @eleurent , CNN works well after your hotfix, many thanks