search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.69k stars 5.73k forks source link

[rllib] attention_net(GTrXLNet) does not support IMPALA algorithm, states are chopped to [BATCH_SIZE / MAX_SEQ_LEN, ...] #10603

Closed forhonourlx closed 1 year ago

forhonourlx commented 4 years ago

What is the problem?

Hi Ray Team, I am running examples/models/attention_net.py with IMPALA, getting the following exceptions: states seem to be chopped into [BATCH_SIZE / MAX_SEQ_LEN, ...], is there any way to fit IMPALA? Could somebody give me a hand? Thanks in advance. Ray version and other system information (Ray master, Python 3.7, TensorFlow 2.3, Ubuntu/Windows):

(pid=26052) 2020-09-05 19:20:42,770        INFO rnn_sequencing.py:110 -- Padded input for RNN:
(pid=26052) Raw batch:
(pid=26052) { 'batch': { 'data': { 'action_dist_inputs': np.ndarray((750, 2), dtype=float64, min=-0.136, max=0.115, mean=0.004),
(pid=26052)                        'action_logp': np.ndarray((750,), dtype=float64, min=-0.815, max=0.0, mean=-0.461),
(pid=26052)                        'action_prob': np.ndarray((500,), dtype=float32, min=0.443, max=0.557, mean=0.502),
(pid=26052)                        'actions': np.ndarray((750,), dtype=float64, min=0.0, max=1.0, mean=0.347),
(pid=26052)                        'agent_index': np.ndarray((500,), dtype=int32, min=0.0, max=0.0, mean=0.0),
(pid=26052)                        'dones': np.ndarray((750,), dtype=float64, min=0.0, max=1.0, mean=0.007),
(pid=26052)                        'eps_id': np.ndarray((500,), dtype=int32, min=634440305.0, max=1765915380.0, mean=982622582.262),
(pid=26052)                        'infos': np.ndarray((500,), dtype=object, head={}),
(pid=26052)                        'obs': np.ndarray((750, 2), dtype=float64, min=0.0, max=1.0, mean=0.333),
(pid=26052)                        'prev_actions': np.ndarray((750,), dtype=float64, min=0.0, max=1.0, mean=0.344),
(pid=26052)                        'prev_rewards': np.ndarray((750,), dtype=float64, min=-1.0, max=1.0, mean=0.043),
(pid=26052)                        'rewards': np.ndarray((750,), dtype=float64, min=-1.0, max=1.0, mean=0.04),
(pid=26052)                        'seq_lens': np.ndarray((15,), dtype=int32, min=1.0, max=50.0, mean=33.333),
(pid=26052)                        'state_in_0': np.ndarray((15, 50, 2), dtype=float32, min=0.0, max=1.0, mean=0.173),
(pid=26052)                        'state_in_1': np.ndarray((15, 50, 64), dtype=float32, min=-0.298, max=0.298, mean=0.003),
(pid=26052)                        'state_in_2': np.ndarray((15, 50, 64), dtype=float32, min=-0.328, max=0.277, mean=0.004),
(pid=26052)                        'state_out_0': np.ndarray((500, 50, 2), dtype=float32, min=0.0, max=1.0, mean=0.373),
(pid=26052)                        'state_out_1': np.ndarray((500, 50, 64), dtype=float32, min=-0.298, max=0.298, mean=0.006),
(pid=26052)                        'state_out_2': np.ndarray((500, 50, 64), dtype=float32, min=-0.332, max=0.282, mean=0.009),
(pid=26052)                        't': np.ndarray((500,), dtype=int32, min=0.0, max=98.0, mean=48.53),
(pid=26052)                        'unroll_id': np.ndarray((500,), dtype=int32, min=0.0, max=14.0, mean=6.53)},
(pid=26052)              'type': 'SampleBatch'},
(pid=26052)   'dynamic_max': True,
(pid=26052)   'features': [ np.ndarray((750,), dtype=float64, min=0.0, max=1.0, mean=0.344),
(pid=26052)                 np.ndarray((750,), dtype=float64, min=-1.0, max=1.0, mean=0.043),
(pid=26052)                 np.ndarray((750, 2), dtype=float64, min=0.0, max=1.0, mean=0.333),
(pid=26052)                 np.ndarray((750, 2), dtype=float64, min=-0.136, max=0.115, mean=0.004),
(pid=26052)                 np.ndarray((750,), dtype=float64, min=-0.815, max=0.0, mean=-0.461),
(pid=26052)                 np.ndarray((750,), dtype=float64, min=0.0, max=1.0, mean=0.347),
(pid=26052)                 np.ndarray((750,), dtype=float64, min=0.0, max=1.0, mean=0.007),
(pid=26052)                 np.ndarray((750, 2), dtype=float64, min=0.0, max=1.0, mean=0.333),
(pid=26052)                 np.ndarray((750,), dtype=float64, min=0.0, max=1.0, mean=0.344),
(pid=26052)                 np.ndarray((750,), dtype=float64, min=-1.0, max=1.0, mean=0.043),
(pid=26052)                 np.ndarray((750,), dtype=float64, min=-1.0, max=1.0, mean=0.04)],
(pid=26052)   'initial_states': [ np.ndarray((15, 50, 2), dtype=float32, min=0.0, max=1.0, mean=0.173),
(pid=26052)                       np.ndarray((15, 50, 64), dtype=float32, min=-0.298, max=0.298, mean=0.003),
(pid=26052)                       np.ndarray((15, 50, 64), dtype=float32, min=-0.328, max=0.277, mean=0.004)],
(pid=26052)   'max_seq_len': 50,
(pid=26052)   'seq_lens': np.ndarray((15,), dtype=int32, min=1.0, max=50.0, mean=33.333),
(pid=26052)   'shuffle': False}
(pid=26052)
(pid=26052) 2020-09-05 19:20:43,828        DEBUG rollout_worker.py:768 -- Training out:
(pid=26052)
(pid=26052) { 'default_policy': { 'learner_stats': { 'cur_lr': 0.0005000000237487257,
(pid=26052)                                          'entropy': 341.80212,
(pid=26052)                                          'entropy_coeff': 0.0010000000474974513,
(pid=26052)                                          'grad_gnorm': 40.0,
(pid=26052)                                          'model': {},
(pid=26052)                                          'policy_loss': 360.0766,
(pid=26052)                                          'var_gnorm': 58.84706,
(pid=26052)                                          'vf_explained_var': 0.0065584183,
(pid=26052)                                          'vf_loss': 4710.1396}}}
(pid=26052) 2020-09-05 19:20:43,831        INFO rnn_sequencing.py:234 -- seq_lens:[50 49  1 50 48  2 50 47  3 50 46  4 50 45  5]
(pid=26052) Exception in thread Thread-1:
(pid=26052) Traceback (most recent call last):
(pid=26052)   File "C:\vnstudio\lib\threading.py", line 917, in _bootstrap_inner
(pid=26052)     self.run()
(pid=26052)   File "C:\vnstudio\lib\site-packages\ray\rllib\execution\learner_thread.py", line 65, in run
(pid=26052)     self.step()
(pid=26052)   File "C:\vnstudio\lib\site-packages\ray\rllib\execution\learner_thread.py", line 72, in step
(pid=26052)     fetches = self.local_worker.learn_on_batch(batch)
(pid=26052)   File "C:\vnstudio\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 765, in learn_on_batch
(pid=26052)     .learn_on_batch(samples)
(pid=26052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\tf_policy.py", line 386, in learn_on_batch
(pid=26052)     fetches = self._build_learn_on_batch(builder, postprocessed_batch)
(pid=26052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\tf_policy.py", line 759, in _build_learn_on_batch
(pid=26052)     self._get_loss_inputs_dict(postprocessed_batch, shuffle=False))
(pid=26052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\tf_policy.py", line 796, in _get_loss_inputs_dict
(pid=26052)     feature_keys=[k for k, v in self._loss_inputs])
(pid=26052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\rnn_sequencing.py", line 94, in pad_batch_to_sequences_of_same_size
(pid=26052)     shuffle=shuffle)
(pid=26052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\rnn_sequencing.py", line 264, in chop_into_sequences
(pid=26052)     s_init.append(s[i])
(pid=26052) IndexError: index 50 is out of bounds for axis 0 with size 15

Reproduction (REQUIRED)

rllib/examples/attention_net.py line17 is changed to: parser.add_argument("--run", type=str, default="IMPALA")

forhonourlx commented 4 years ago

Hi @sven1977 According to https://github.com/ray-project/ray/issues/8616 IMPALA crashes if using an LSTM and num_sgd_iter > 1 When using IMPALA in attention_net, if num_sgd_iter > 1, it crashes. And PPO will not. initial_states seem to be chopped into [BATCH_SIZE / MAX_SEQ_LEN, ...], how does num_sgd_iter effect initial_states?

(pid=7052) 2020-09-06 05:09:19,537 INFO rnn_sequencing.py:107 -- Padded input for RNN:
(pid=7052)
(pid=7052) { 'features': [ np.ndarray((1000,), dtype=float64, min=0.0, max=1.0, mean=0.542),
(pid=7052)                 np.ndarray((1000,), dtype=float64, min=-1.0, max=1.0, mean=0.028),
(pid=7052)                 np.ndarray((1000, 2), dtype=float64, min=0.0, max=1.0, mean=0.5),
(pid=7052)                 np.ndarray((1000, 2), dtype=float64, min=-0.206, max=0.521, mean=0.112),
(pid=7052)                 np.ndarray((1000,), dtype=float64, min=-0.898, max=-0.523, mean=-0.684),
(pid=7052)                 np.ndarray((1000,), dtype=float64, min=0.0, max=1.0, mean=0.554),
(pid=7052)                 np.ndarray((1000,), dtype=float64, min=0.0, max=0.0, mean=0.0),
(pid=7052)                 np.ndarray((1000, 2), dtype=float64, min=0.0, max=1.0, mean=0.5),
(pid=7052)                 np.ndarray((1000,), dtype=float64, min=0.0, max=1.0, mean=0.542),
(pid=7052)                 np.ndarray((1000,), dtype=float64, min=-1.0, max=1.0, mean=0.028),
(pid=7052)                 np.ndarray((1000,), dtype=float64, min=-1.0, max=1.0, mean=0.026)],
(pid=7052)   'initial_states': [ np.ndarray((20, 50, 2), dtype=float32, min=0.0, max=0.0, mean=0.0),
(pid=7052)                       np.ndarray((20, 50, 64), dtype=float32, min=0.0, max=0.0, mean=0.0)],
(pid=7052)   'max_seq_len': 50,
(pid=7052)   'seq_lens': np.ndarray((20,), dtype=int32, min=50.0, max=50.0, mean=50.0)}
(pid=7052)
(pid=7052) Exception in thread Thread-1:
(pid=7052) Traceback (most recent call last):
(pid=7052)   File "C:\vnstudio\lib\threading.py", line 917, in _bootstrap_inner
(pid=7052)     self.run()
(pid=7052)   File "C:\vnstudio\lib\site-packages\ray\rllib\execution\learner_thread.py", line 65, in run
(pid=7052)     self.step()
(pid=7052)   File "C:\vnstudio\lib\site-packages\ray\rllib\execution\learner_thread.py", line 72, in step
(pid=7052)     fetches = self.local_worker.learn_on_batch(batch)
(pid=7052)   File "C:\vnstudio\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 757, in learn_on_batch
(pid=7052)     .learn_on_batch(samples)
(pid=7052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\tf_policy.py", line 385, in learn_on_batch
(pid=7052)     fetches = self._build_learn_on_batch(builder, postprocessed_batch)
(pid=7052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\tf_policy.py", line 759, in _build_learn_on_batch
(pid=7052)     self._get_loss_inputs_dict(postprocessed_batch, shuffle=False))
(pid=7052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\tf_policy.py", line 796, in _get_loss_inputs_dict
(pid=7052)     feature_keys=[k for k, v in self._loss_inputs])
(pid=7052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\rnn_sequencing.py", line 94, in pad_batch_to_sequences_of_same_size
(pid=7052)     shuffle=shuffle)
(pid=7052)   File "C:\vnstudio\lib\site-packages\ray\rllib\policy\rnn_sequencing.py", line 254, in chop_into_sequences
(pid=7052)     s_init.append(s[i])
(pid=7052) IndexError: index 50 is out of bounds for axis 0 with size 20
zzchuman commented 4 years ago

Hi, ray team! I am running examples/attention_net.py . @sven1977 , @ericl I input the python /examples /attention_net.py . But,

image

zzchuman commented 4 years ago

hello,@ericl @sven1977 I read the error.txt. image

It means You must feed a value for placeholder tensor 'default_policy.

I should write the default policy,Right?