i get error when i use elon example

[wac81]

Traceback (most recent call last): File "/data/TextRL/train2.py", line 46, in pfrl.experiments.train_agent_with_evaluation( File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/experiments/train_agent.py", line 208, in train_agent_with_evaluation eval_stats_history = train_agent( File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/experiments/train_agent.py", line 57, in train_agent action = agent.act(obs) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/agent.py", line 161, in act return self.batch_act([obs])[0] File "/data/TextRL/textrl/actor.py", line 216, in batch_act return self._batch_act_train(batch_obs) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/agents/ppo.py", line 735, in _batch_act_train action_distrib, batch_value = self.model(b_state) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/nn/branched.py", line 30, in forward return tuple(mod(*args, *kwargs) for mod in self.child_modules) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/nn/branched.py", line 30, in return tuple(mod(args, kwargs) for mod in self.child_modules) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/container.py", line 204, in forward input = module(input) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "/data/TextRL/env/lib/python3.8/site-packages/accelerate/hooks.py", line 158, in new_forward output = old_forward(args, kwargs) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: expected scalar type Float but found BFloat16

[voidful]

I tested on Colab and everything worked fine. It looks like you're using bf16. May I know what model you're using?

[wac81]

i use this model： checkpoint = "bigscience/bloom-560m"

[wac81]

and if i use gp2, i get new ERROR like this: actions = torch.tensor([b["action"] for b in dataset], device=device) Traceback (most recent call last): File "/home/wac/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/wac/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/wac/.vscode-server/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/main.py", line 45, in cli.main() File "/home/wac/.vscode-server/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main run() File "/home/wac/.vscode-server/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file runpy.run_path(target_as_str, run_name=compat.force_str("main")) File "/home/wac/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "/home/wac/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/home/wac/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/data/TextRL/train_bloom.py", line 47, in agent.observe(obs, reward, done, reset) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/agent.py", line 164, in observe self.batch_observe([obs], [reward], [done], [reset]) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/agents/ppo.py", line 684, in batch_observe self._batch_observe_train(batch_obs, batch_reward, batch_done, batch_reset) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/agents/ppo.py", line 810, in _batch_observe_train self._update_if_dataset_is_ready() File "/data/TextRL/textrl/actor.py", line 194, in _update_if_dataset_is_ready self._update(dataset) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/agents/ppo.py", line 490, in _update distribs, vs_pred = self.model(states) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/nn/branched.py", line 30, in forward return tuple(mod(*args, *kwargs) for mod in self.child_modules) File "/data/TextRL/env/lib/python3.8/site-packages/pfrl/nn/branched.py", line 30, in return tuple(mod(args, kwargs) for mod in self.child_modules) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/container.py", line 204, in forward input = module(input) File "/data/TextRL/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(input, **kwargs) File "/data/TextRL/textrl/actor.py", line 153, in forward return torch.distributions.Categorical(logits=logits) File "/data/TextRL/env/lib/python3.8/site-packages/torch/distributions/categorical.py", line 66, in init super(Categorical, self).init(batch_shape, validate_args=validate_args) File "/data/TextRL/env/lib/python3.8/site-packages/torch/distributions/distribution.py", line 56, in init raise ValueError( ValueError: Expected parameter logits (Tensor of shape (3, 2, 50257)) of distribution Categorical(logits: torch.Size([3, 2, 50257])) to satisfy the constraint IndependentConstraint(Real(), 1), but found invalid values: tensor([[[nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan]],

    [[nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan]],

    [[nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan]]], device='cuda:0',
   grad_fn=<SubBackward0>)

[wac81]

i follow your elon musk example.

[voidful]

be careful to the learning rate when fine-tuning via RL, setting a lower learning rate should be helpful here is the colab example, both model are working:

colab example: bigscience/bloom-560m

colab exmaple: huggingtweets/elonmusk

[wac81]

thank you, i found out my error from load model with causlLM loader