[QuestionGeneration] RuntimeError: Integer division of tensors using div or / is no longer supported

jqueguiner commented 2 years ago

Environment info

transformers version: 4.12.0
Platform: Linux-5.4.0-88-generic-x86_64-with-debian-buster-sid
Python version: 3.7.10
PyTorch version (GPU?): 1.6.0 (True)
Tensorflow version (GPU?): 2.6.0 (False)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: No
Using distributed or parallel set-up in script?: No

Who can help

Models:

T5ForConditionalGeneration : @patrickvonplaten, @patil-suraj

Information

Model I am using T5conditionalGeneration through Questgen.ai

The problem arises when using:

[x] my own modified scripts: (give details below)


from transformers import AutoTokenizer, AutoModelWithLMHead

def predict(sentence): tokenizer = AutoTokenizer.from_pretrained("flexudy/t5-base-multi-sentence-doctor")

model = AutoModelWithLMHead.from_pretrained("flexudy/t5-base-multi-sentence-doctor")

input_text = f"repair_sentence: {sentence}</s>"

input_ids = tokenizer.encode(input_text, return_tensors="pt")

outputs = model.generate(input_ids, max_length=32, num_beams=1)

sentence = tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)

return sentence


The tasks I am working on is:
* [x] my own task : 
```python
  File "apis/text/text/boolean-question-generations/questgen/questgen.py", line 12, in predict
    output = qe.predict_boolq(payload)
  File "/opt/conda/lib/python3.7/site-packages/Questgen/main.py", line 238, in predict_boolq
    output = beam_search_decoding (input_ids, attention_masks,self.model,self.tokenizer)
  File "/opt/conda/lib/python3.7/site-packages/Questgen/encoding/encoding.py", line 18, in beam_search_decoding
    early_stopping=True
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/generation_utils.py", line 1064, in generate
    **model_kwargs,
  File "/opt/conda/lib/python3.7/site-packages/transformers/generation_utils.py", line 1839, in beam_search
    next_indices = (next_tokens / vocab_size).long()
RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.

To reproduce

Steps to reproduce the behavior:

1.

pip install git+https://github.com/ramsrigouthamg/Questgen.ai
pip install git+https://github.com/boudinfl/pke.git

python -m nltk.downloader universal_tagset
python -m spacy download en

2.

from transformers import AutoTokenizer, AutoModelWithLMHead

def predict(sentence):
    tokenizer = AutoTokenizer.from_pretrained("flexudy/t5-base-multi-sentence-doctor")

    model = AutoModelWithLMHead.from_pretrained("flexudy/t5-base-multi-sentence-doctor")

    input_text = f"repair_sentence: {sentence}</s>"

    input_ids = tokenizer.encode(input_text, return_tensors="pt")

    outputs = model.generate(input_ids, max_length=32, num_beams=1)

    sentence = tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)

    return sentence

Expected behavior

No error

Fix found

https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L1839

use // instead of /

jqueguiner commented 2 years ago

oups, I guess we are all colliding here here is the related change 3months ago that removed the // operator because of torch working. https://github.com/huggingface/transformers/pull/13013

my 2 cents is that implicit operator are kind of broken

I feel like https://github.com/StevenTang1998 push toward explicit operation https://github.com/huggingface/transformers/pull/13013#issuecomment-894198148 but not supported for torch < 1.8

jqueguiner commented 2 years ago

@sgugger @nreimers @patrickvonplaten

any thoughs on that matter ?

sgugger commented 2 years ago

I think we will need to do a version check and use @StevenTang1998 solution for recent versions of PyTorch and leave the old code for older versions.

Do you want to take a stab at a PR?

jqueguiner commented 2 years ago

I could try to write it if you want.

sgugger commented 2 years ago

I think I just asked you to ;-)

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

JeetRoy97 commented 2 years ago

I am still getting the error in generation_utils.py. The issue is not yet resolved in Transformers 4.15.0

patrickvonplaten commented 2 years ago

@JeetRoy97,

Could you also post your environment info?

Just run transformes-cli env and copy paste the output here. Especially your Python and PyTorch versions would be important to know

jx669 commented 2 years ago

I run into the same division error torch: 1.6.0 transformers: 4.15.0 onnxruntime: 1.10.0 python: 3.6 GPU

tokenizer =M2M100Tokenizer.from_pretrained( path ) tokenizer.src_lang ='xx' example =['xxxx'] encoded = tokenizer(example, return_tensors='pt') generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.get_lang_id('en'))

Error occurred at the model.generatestep. error message was directed to "transformers/generation_utils.py" line 1955: next_indices = (next_tokens / vocab_size).long()

I am not entirely sure if you can reproduce because I used the onnx version of the M2M model.

This one looks like a similar problem: https://discuss.pytorch.org/t/runtimeerror-integer-division-of-tensors-using-div-or-is-no-longer-supported-and-in-a-future-release-div-will-perform-true-division-as-in-python-3-use-true-divide-or-floor-divide-in-python-instead/99427 and it was solved by changing the input type image = image.float() , but in the case of NLP transformer, the inputs are ids. I tried to use encoded = encoded.to(torch.long), no luck.

Any advice would be highly appreciated!

mwojnars commented 2 years ago

Same error when running the fine-tuning script, run_summarization.py with --predict_with_generate. The stack trace:

Traceback (most recent call last):
  File "run_summarization.py", line 698, in <module>
    main()
  File "run_summarization.py", line 650, in main
    predict_results = trainer.predict(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer_seq2seq.py", line 119, in predict
    return super().predict(test_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2319, in predict
    output = eval_loop(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2419, in evaluation_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer_seq2seq.py", line 172, in prediction_step
    generated_tokens = self.model.generate(
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 1239, in generate
    return self.beam_search(
  File "/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 2027, in beam_search
    next_indices = (next_tokens / vocab_size).long()
RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.

Environment:

- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

patrickvonplaten commented 2 years ago

Hey @mwojnars,

Thanks for reporting the error here. I'm attaching a PR that should fix it.

mwojnars commented 2 years ago

@patrickvonplaten Thanks. It seems to be working fine now.

janspoerer commented 2 years ago

The error seems to persist when running the RAG-Token model: https://huggingface.co/facebook/rag-token-nq. Running the exact code from the aforementioned link with a fresh install of the transformers library yields the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_35233/3517742745.py in <module>
      1 input_dict = tokenizer.prepare_seq2seq_batch("who holds the record in 100m freestyle", return_tensors="pt")
      2 
----> 3 generated = model.generate(input_ids=input_dict["input_ids"])
      4 print(tokenizer.batch_decode(generated, skip_special_tokens=True)[0])
      5 

~/anaconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17 

~/anaconda3/lib/python3.8/site-packages/transformers/models/rag/modeling_rag.py in generate(self, input_ids, attention_mask, context_input_ids, context_attention_mask, doc_scores, max_length, min_length, early_stopping, use_cache, num_beams, num_beam_groups, diversity_penalty, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, repetition_penalty, bad_words_ids, num_return_sequences, decoder_start_token_id, n_docs, prefix_allowed_tokens_fn, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, **model_kwargs)
   1614                 num_beam_hyps_to_keep=num_return_sequences,
   1615             )
-> 1616             return self.beam_search(
   1617                 input_ids,
   1618                 beam_scorer,

~/anaconda3/lib/python3.8/site-packages/transformers/generation_utils.py in beam_search(self, input_ids, beam_scorer, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, **model_kwargs)
   1853             )
   1854 
-> 1855             next_indices = (next_tokens / vocab_size).long()
   1856             next_tokens = next_tokens % vocab_size
   1857 

RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.

and the following warnings:

/home/myname/anaconda3/lib/python3.8/site-packages/transformers/models/rag/tokenization_rag.py:92: FutureWarning: `prepare_seq2seq_batch` is deprecated and will be removed in version 5 of 🤗 Transformers. Use the regular `__call__` method to prepare your inputs and the tokenizer under the `with_target_tokenizer` context manager to prepare your targets. See the documentation of your specific tokenizer for more details
  warnings.warn(
/home/myname/anaconda3/lib/python3.8/site-packages/transformers/generation_utils.py:1747: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
  warnings.warn(

I cannot rule out that this is an error specific to my installation. But as I've just installed the transformers library, I suspect that this might be a more general issue.

huggingface / transformers