Error Restoring PPO model with Custom Head

[RaedShabbir]

What is the problem?

I have used tune to optimize and train PPO with a parametric head similar to the example seen here , however, when trying to restore my trained model with rllib for testing I get the following error. I am still able to continue training via tune, but cannot restore the model using the rllib standard procedure.

Ray version and other system information (Python version, TensorFlow version, OS): ray == 0.9.0.dev0 tensorflow == 2.0.0 Python 3.6.9 :: Anaconda, Inc.

Reproduction (REQUIRED)

Unfortunatley I can't provide a full script as of now, here is a snippet of the error and the call to restore.

`test_config['model'] = self._config['tune']['model'] test_config['model']['custom_options']['true_obs_shape'] = (self._config['environment']['window_size'], self._config['environment']["num_features"]) test_config['model']['custom_options']['action_embed_size'] = self._config['environment']['embedding_size']

    #path to pretrained model 
    checkpoint = self.get_checkpoint()

    #create model 
    model = self._config['trainer'](env="TradingEnvironment", config=test_config)
    model.restore(checkpoint)`

AttributeError: 'list' object has no attribute 'keys' lib/python3.6/site-packages/ray/experimental/tf_utils.py", line 182, in set_weights self.assignment_nodes[name] for name in new_weights.keys()

From here `def set_weights(self, new_weights): """Sets the weights to new_weights.

    Note:
        Can set subsets of variables as well, by only passing in the
        variables you want to be set.

    Args:
        new_weights (Dict): Dictionary mapping variable names to their
            weights.
    """
    self._check_sess()
    assign_list = [
        self.assignment_nodes[name] for name in new_weights.keys()``

[sven1977]

Sorry, but I would need some minimal script, where I can debug this. Maybe use our rllib/examples/random_env.py::RandomEnv with your particular action-space and provide a minimal model where this would break.

[fhabermacher]

Not sure this helps, but I get the same AttributeError for self.assignment_nodes[name] for name in new_weights.keys() when simply run the 'official' ray/rllib RL Exercise 2 - Proximal Policy Optimization but adding a tf.compat.v1.enable_v2_behavior() statement before the ray.init statement.

Ray version and other system information (Python version, TensorFlow version, OS): ray == 0.8.2 tensorflow == 2.1.0 Python 3.6.10 OS: Ubuntu 16.04

[athenawisdoms]

@ericl @sven1977 Here's a script to reproduce the error

import gym
import ray
from ray.rllib.agents.ppo import PPOTrainer, DEFAULT_CONFIG
from ray.tune.logger import pretty_print

import tensorflow as tf
tf.compat.v1.enable_v2_behavior()

ray.init(num_cpus=3, ignore_reinit_error=True, log_to_driver=False)

config = DEFAULT_CONFIG.copy()
config['num_workers'] = 2
config['num_sgd_iter'] = 30
config['sgd_minibatch_size'] = 128
config['model']['fcnet_hiddens'] = [100, 100]
config['num_cpus_per_worker'] = 0  # This avoids running out of resources in the notebook environment when this cell is re-executed

agent = PPOTrainer(config, 'CartPole-v0')

for i in range(2):
    result = agent.train()
    print(pretty_print(result))

gives the error messages

2020-05-31 01:17:27,640 INFO resource_spec.py:204 -- Starting Ray with 23.49 GiB memory available for workers and up to 11.75 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-05-31 01:17:27,925 INFO services.py:1168 -- View the Ray dashboard at localhost:8265
2020-05-31 01:17:28,493 INFO trainer.py:578 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
2020-05-31 01:17:28,607 INFO trainable.py:217 -- Getting current IP.
2020-05-31 01:17:28,607 WARNING util.py:37 -- Install gputil for GPU system monitoring.
2020-05-31 01:17:35,867 ERROR worker.py:1029 -- Possible unhandled error from worker: ray::RolloutWorker.set_weights() (pid=31979, ip=172.26.51.162)
  File "python/ray/_raylet.pyx", line 463, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 417, in ray._raylet.execute_task.function_executor
  File "/home/athena/miniconda3/envs/prettyanimals/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 571, in set_weights
    self.policy_map[pid].set_weights(w)
  File "/home/athena/miniconda3/envs/prettyanimals/lib/python3.8/site-packages/ray/rllib/policy/tf_policy.py", line 357, in set_weights
    return self._variables.set_weights(weights)
  File "/home/athena/miniconda3/envs/prettyanimals/lib/python3.8/site-packages/ray/experimental/tf_utils.py", line 182, in set_weights
    self.assignment_nodes[name] for name in new_weights.keys()
AttributeError: 'list' object has no attribute 'keys'
2020-05-31 01:17:35,868 ERROR worker.py:1029 -- Possible unhandled error from worker: ray::RolloutWorker.set_weights() (pid=31977, ip=172.26.51.162)
  File "python/ray/_raylet.pyx", line 463, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 417, in ray._raylet.execute_task.function_executor
  File "/home/athena/miniconda3/envs/prettyanimals/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 571, in set_weights
    self.policy_map[pid].set_weights(w)
  File "/home/athena/miniconda3/envs/prettyanimals/lib/python3.8/site-packages/ray/rllib/policy/tf_policy.py", line 357, in set_weights
    return self._variables.set_weights(weights)
  File "/home/athena/miniconda3/envs/prettyanimals/lib/python3.8/site-packages/ray/experimental/tf_utils.py", line 182, in set_weights
    self.assignment_nodes[name] for name in new_weights.keys()
AttributeError: 'list' object has no attribute 'keys'

Environment

• ray 0.8.5
• Python 3.8.3
• tensorflow 2.2.0
• Ubuntu 18.04 in WSL2

[Maltimore]

see also #8993 which seems to be the same error. (By the way, I'm also getting this error and I'm not importing tensorflow at all, in fact I use config["framework"] = "pytorch"). edit: nevermind, I should've used "torch"

[stale[bot]]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale[bot]]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!