[rllib] [tune] Cannot create fcnet when using HyperOptSearch

[ghost]

What is the problem?

I am working with DQN of rllib and HyperOptSearch of tune. I found that there is something wrong with creating fcnet.

When I run the code (at the bottom), it throws an error. Here is a brief of error message:

  File ".../ray/rllib/agents/dqn/dqn_torch_policy.py", line 153, in build_q_model_and_distribution
    num_outputs = ([256] + config["model"]["fcnet_hiddens"])[-1]
TypeError: can only concatenate list (not "tuple") to list

After digging a bit into the code I find that the value of hyperparameter fcnet_hiddens become a tuple object after some operations of HyperOptSearch, though it was a list object. So, config["model"]["fcnet_hiddens"] (tuple object) cannot concatenate to [256] (list object). If I don't use HyperOptSearch (i.e., normal FIFO scheduler instead), there will be nothing wrong.

To deal with this issue, I tried to set hiddens (a hyperparameter of Dueling DQN) to [] instead of [256] (which is DQN default setting). The issue above was then solved but another similar issue occurred:

  File ".../ray/rllib/models/torch/fcnet.py", line 28, in __init__
    model_config.get("post_fcnet_hiddens", [])
TypeError: can only concatenate tuple (not "list") to tuple

This time, it is because post_fcnet_hiddens is a empty list object by default and fcnet_hiddens is still a tuple object. To deal with it, I tried to set post_fcnet_hiddens to an empty tuple .

Up to now, the two issues are solved via some redundant settings. But I think it would be better to make some changes in the source code to avoid these issues.

BTW, these issues occur no matter how I set dueling (True or False). But I think if user set dueling to False, the hyperparameter hiddens won't influence anything, isn't right?

Ray version and other system information (Python version, TensorFlow version, OS): Python version: 3.7.9 Pytorch version: 1.7.1 TensorFlow version: 2.4.1 OS: Ubuntu 20.04.2 LTS

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

import ray
from ray.rllib.agents.dqn import dqn
from ray import tune
from ray.tune.suggest.hyperopt import HyperOptSearch

if __name__ == "__main__":
    ray.init(local_mode=True)

    config = {
        "framework": "torch",

        "env": "CartPole-v1",
        "dueling": True,
        # "hiddens": [],

        "model": {
            "fcnet_hiddens": [tune.randint(8, 256)],
            # "post_fcnet_hiddens": tuple(),
        },
    }

    HyperOpt_algo = HyperOptSearch()

    analysis = tune.run(
        dqn.DQNTrainer,
        search_alg=HyperOpt_algo,
        config=config,
        metric="episode_reward_mean",
        mode="max",
        num_samples=10,
    )

    ray.shutdown()

If the code snippet cannot be run by itself, the issue will be closed with "needs-repro-script".

• [x] I have verified my script runs in a clean environment and reproduces the issue.
• [x] I have verified the issue also occurs with the latest wheels.

[krfricke]

Thanks for posting this.

Converting list spaces to tuples is an issue of upstream hyperopt (though I guess semantically it makes sense, as a tuple is just an immutable sequence).

However, #14308 addresses this on our end by casting the config object to a list before doing list-related operations.

Thanks for raising this issue!

[ghost]

Thanks for quick reply!