The second time I run the code, it raises an error. After some investigation, I found that the problem is with the forward pass, which substitutes the token in input_ids with the correct image token from the autoencoder. However, when run the second time, it attempts to perform the substitution again since pixel_values are still present, but the token is no longer there (as it has been replaced with the real image tokens).
Expected behavior
I expect the forward pass not to modify the input tensors.
System Info
transformers
version: 4.43.0.dev0Who can help?
@zucchini-nlp
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
After importing the model and processor
Execute
The second time I run the code, it raises an error. After some investigation, I found that the problem is with the forward pass, which substitutes the token in input_ids with the correct image token from the autoencoder. However, when run the second time, it attempts to perform the substitution again since pixel_values are still present, but the token is no longer there (as it has been replaced with the real image tokens).
Expected behavior
I expect the forward pass not to modify the input tensors.