search

HHousen / TransformerSum

Models to perform neural summarization (extractive and abstractive) using machine learning transformers and a tool to convert abstractive summarization datasets to the extractive task.
https://transformersum.rtfd.io
GNU General Public License v3.0
429 stars 58 forks source link

"TypeError: forward() got an unexpected keyword argument 'use_cache'" for AbstractiveSummarizer #48

Closed JoachimJaafar closed 3 years ago

JoachimJaafar commented 3 years ago

Hi,

I'm trying to run the following command (from https://transformersum.readthedocs.io/en/latest/abstractive/training.html, I just added --gpus 0) :

python main.py --mode abstractive --model_name_or_path bert-base-uncased --decoder_model_name_or_path bert-base-uncased --cache_file_path data --max_epochs 4 --do_train --do_test --batch_size 4 --weights_save_path model_weights --no_wandb_logger_log_model --accumulate_grad_batches 5 --use_scheduler linear --warmup_steps 8000 --gradient_clip_val 1.0 --custom_checkpoint_every_n 300 --gpus 0

Here's is my environment.yml :

name: transformersum
channels:
    - conda-forge
    - pytorch
dependencies:
    - pytorch
    - scikit-learn
    - tensorboard
    - spacy
    - spacy-model-en_core_web_sm
    - sphinx
    - pyarrow
    - pip
    - pip:
      - pytorch_lightning
      - transformers
      - torch_optimizer
      - click==7.0
      - wandb
      - rouge-score
      - packaging
      - datasets
      - gradio
      - tokenizers==0.8.0rc4
variables:
    TOKENIZERS_PARALLELISM: true

When performing the validation sanity check, I get the following exception :

Traceback (most recent call last):
  File "main.py", line 457, in <module>
    main(main_args)
  File "main.py", line 119, in main
    trainer.fit(model)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
    self.dispatch()
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
    self.accelerator.start_training(self)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
    return self.run_train()
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in run_train
    self.run_sanity_check(self.lightning_module)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1107, in run_sanity_check
    self.run_evaluation()
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in run_evaluation
    output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 174, in evaluation_step
    output = self.trainer.accelerator.validation_step(args)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 226, in validation_step
    return self.training_type_plugin.validation_step(*args)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in validation_step
    return self.lightning_module.validation_step(*args, **kwargs)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 701, in validation_step
    cross_entropy_loss = self._step(batch)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 686, in _step
    outputs = self.forward(source, target, source_mask, target_mask, labels=labels)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 248, in forward
    **kwargs
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py", line 276, in forward
    **kwargs_encoder,
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'use_cache'

I'm currently using Python 3.6.13. Do you have an idea on what I'm doing wrong or need to change in any of my files ? Thanks !

HHousen commented 3 years ago

@JoachimJaafar I think you might have an old version of huggingface/transformers because this file transformers/modeling_encoder_decoder.py shown in the error message used to be at that path but was moved a while ago in an update. Try upgrading with pip install -U transformers (or using conda). Maybe this line tokenizers==0.8.0rc4 is blocking the new version from installing? Let me know if this solves the problem.

JoachimJaafar commented 3 years ago

@HHousen After upgrading and removing tokenizers==0.8.0rc4, I get the following exception, still when performing the validation sanity check :

Traceback (most recent call last):
  File "main.py", line 457, in <module>
    main(main_args)
  File "main.py", line 119, in main
    trainer.fit(model)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
    self.dispatch()
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
    self.accelerator.start_training(self)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
    return self.run_train()
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in run_train
    self.run_sanity_check(self.lightning_module)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1107, in run_sanity_check
    self.run_evaluation()
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in run_evaluation
    output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 174, in evaluation_step
    output = self.trainer.accelerator.validation_step(args)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 226, in validation_step
    return self.training_type_plugin.validation_step(*args)
  File "/opt/conda/envs/transformersum/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in validation_step
    return self.lightning_module.validation_step(*args, **kwargs)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 701, in validation_step
    cross_entropy_loss = self._step(batch)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 686, in _step
    outputs = self.forward(source, target, source_mask, target_mask, labels=labels)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 254, in forward
    loss = self.calculate_loss(prediction_scores, labels)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 666, in calculate_loss
    prediction_scores.view(-1, self.model.config.vocab_size), labels.view(-1)
AttributeError: 'EncoderDecoderConfig' object has no attribute 'vocab_size'
HHousen commented 3 years ago

@JoachimJaafar According to the Configuration documentation the vocab_size attribute is present in all subclasses. For example, this encoder_decoder config.json file shows that the vocab_size is present. It is possible that the vocab_size is only stored in the configuration for the encoder or decoder. So, try replacing line 666 (prediction_scores.view(-1, self.model.config.vocab_size), labels.view(-1)) to prediction_scores.view(-1, self.model.config.encoder.vocab_size), labels.view(-1). If this works then I will change the code to use the right key.

JoachimJaafar commented 3 years ago

It works now, thanks for your help !

JoachimJaafar commented 3 years ago

Actually, after I deleted the data to try to download it again, I got the following exception : 

Traceback (most recent call last):
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/main.py", line 457, in <module>
    main(main_args)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/main.py", line 119, in main
    trainer.fit(model)
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 704, in _run
    self.data_connector.prepare_data(model)
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 57, in prepare_data
    model.prepare_data()
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 541, in prepare_data
    self.dataset[split] = self.dataset[split].map(
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1590, in map
    return self._map_single(
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 176, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1930, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1826, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "/root/project/ia-etat-de-l-art/mlflow/scripts/TransformerSum/src/abstractive.py", line 385, in convert_to_features
    sents_input_ids = self.tokenizer.encode_plus(
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2378, in encode_plus
    return self._encode_plus(
  File "/opt/conda/envs/transformersum/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 458, in _encode_plus
    batched_output = self._batch_encode_plus(
TypeError: _batch_encode_plus() got an unexpected keyword argument 'is_pretokenized'
HHousen commented 3 years ago

Whoops, good catch. It looks like that attribute was renamed when huggingface/transformers upgraded recently: "The tokenizer encoding argument is_pretokenized becomes is_split_into_words" from https://huggingface.co/transformers/migration.html?highlight=is_pretokenized. Should be fixed in 03eeee83b8705866c358a13ad70838425eefe6b7.