ModelCatalog does not support model interface when using pytorch

[BigRiceBall-zz]

Problem Description

I try to implement the DQN with pytorch. The get_model_v2 in ModelCatalog would call model interface so it can create Q head if the framework is tensorflow. But the model interface is never called if the framework is pytorch.

Source code

Tensorflow:

            legacy_model_cls = default_model or ModelCatalog.get_model  
            wrapper = ModelCatalog._wrap_if_needed(
                make_v1_wrapper(legacy_model_cls), model_interface)
            return wrapper(obs_space, action_space, num_outputs, model_config,
                           name, **model_kwargs)

Pytorch:

            if default_model:
                return default_model(obs_space, action_space, num_outputs,
                                     model_config, name)
            return ModelCatalog._get_default_torch_model_v2(
                obs_space, action_space, num_outputs, model_config, name)

[sytelus]

I can confirm this. When trying to use PyTorch model with DQN, currently the following error is raised. For example, when trying to use VisionNetwork from torch we get:

  File "/home/shitals/GitHubSrc/ray/python/ray/rllib/agents/dqn/dqn_policy.py", line 215, in build_q_model
    parameter_noise=config["parameter_noise"])
  File "/home/shitals/GitHubSrc/ray/python/ray/rllib/models/catalog.py", line 263, in get_model_v2
    model_interface)
ValueError: ('The given model must subclass', <class 'ray.rllib.agents.dqn.distributional_q_model.DistributionalQModel'>)

It seems TF is hardcoded in dql_policy.py.

[stale[bot]]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale[bot]]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!