search

richarddwang / electra_pytorch

Pretrain and finetune ELECTRA with fastai and huggingface. (Results of the paper replicated !)
324 stars 41 forks source link

Error in store_attr() #2

Closed glakshmidhar closed 4 years ago

glakshmidhar commented 4 years ago

Tried running the pretrain.py script when I got this error

process id: 76202
{'device': 'cuda:0', 'base_run_name': 'vanilla', 'seed': 11081, 'adam_bias_correction': False, 'schedule': 'original_linear', 'sampling': 'fp32_gumbel', 'electra_mask_style': True, 'tie_gen_in_out_embedding': False, 'gen_smooth_label': False, 'disc_smooth_label': False, 'size': 'small', 'my_model': False, 'run_name': 'vanilla_11081', 'mask_prob': 0.15, 'lr': 0.0005, 'bs': 128, 'steps': 1000000, 'max_length': 128}
{}
loading the electra data (wiki)
loading the electra data (BookCorpus)
electra_pytorch/venv/lib/python3.6/site-packages/nlp/utils/py_utils.py:191: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return function(data_struct)
Traceback (most recent call last):
  File "pretrain.py", line 390, in <module>
    RunSteps(c.steps, [0.0625, 0.125, 0.25, 0.5, 1.0], c.run_name+"_{percent}"),
  File "electra_pytorch/electra_pytorch/_utils/would_like_to_pr.py", line 51, in __init__
    store_attr(self, 'n_steps,save_points,base_name,no_val')
  File "electra_pytorch/venv/lib/python3.6/site-packages/fastcore/utils.py", line 97, in store_attr
    if not hasattr(self, '__stored_args__'): self.__stored_args__ = {}
AttributeError: 'str' object has no attribute '__stored_args__'

Updating fastai and fastcore does not solve the issue. I even when through the documentation and found that if we don't provide arguments to store_attr(), it would automatically store all parameters passed to the function. I tried that and the model did start training, but it got nan loss right from epoch 1. Could you please help me resolve this? @richarddwang 

process id: 76772
{'device': 'cuda:0', 'base_run_name': 'vanilla', 'seed': 11081, 'adam_bias_correction': False, 'schedule': 'original_linear', 'sampling': 'fp32_gumbel', 'electra_mask_style': True, 'tie_gen_in_out_embedding': False, 'gen_smooth_label': False, 'disc_smooth_label': False, 'size': 'small', 'my_model': False, 'run_name': 'vanilla_11081', 'mask_prob': 0.15, 'lr': 0.0005, 'bs': 128, 'steps': 1000000, 'max_length': 128}
{}
loading the electra data (wiki)
loading the electra data (BookCorpus)
electra_pytorch/venv/lib/python3.6/site-packages/nlp/utils/py_utils.py:191: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return function(data_struct)
vanilla_11081 , starts at 2020-09-07 09:02:02.063122
epoch     train_loss  valid_loss  time
0         nan         00:13
1         nan         00:18
richarddwang commented 4 years ago

Hi @glakshmidhar , Thanks for the report. I commit the fix to adapt the api change of store_attr and I forgot to push it. 😂 Please reopen the issue if you still get the same problem.