sgrvinod / a-PyTorch-Tutorial-to-Image-Captioning

Show, Attend, and Tell | a PyTorch Tutorial to Image Captioning
MIT License
2.76k stars 713 forks source link

same caption for every image #110

Closed purveshpatel511 closed 4 years ago

purveshpatel511 commented 4 years ago

i get error after running caption.py file:::

/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'models.Encoder' has changed. Saved a reverse patch to Encoder.patch. Run `patch -p0 < Encoder.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.container.Sequential' has changed. Saved a reverse patch to Sequential.patch. Run `patch -p0 < Sequential.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.conv.Conv2d' has changed. Saved a reverse patch to Conv2d.patch. Run `patch -p0 < Conv2d.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm2d' has changed. Saved a reverse patch to BatchNorm2d.patch. Run `patch -p0 < BatchNorm2d.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.activation.ReLU' has changed. Saved a reverse patch to ReLU.patch. Run `patch -p0 < ReLU.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.pooling.MaxPool2d' has changed. Saved a reverse patch to MaxPool2d.patch. Run `patch -p0 < MaxPool2d.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torchvision.models.resnet.Bottleneck' has changed. Saved a reverse patch to Bottleneck.patch. Run `patch -p0 < Bottleneck.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.pooling.AdaptiveAvgPool2d' has changed. Saved a reverse patch to AdaptiveAvgPool2d.patch. Run `patch -p0 < AdaptiveAvgPool2d.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'models.DecoderWithAttention' has changed. Saved a reverse patch to DecoderWithAttention.patch. Run `patch -p0 < DecoderWithAttention.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'models.Attention' has changed. Saved a reverse patch to Attention.patch. Run `patch -p0 < Attention.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. Saved a reverse patch to Linear.patch. Run `patch -p0 < Linear.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Softmax' has changed. Saved a reverse patch to Softmax.patch. Run `patch -p0 < Softmax.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.sparse.Embedding' has changed. Saved a reverse patch to Embedding.patch. Run `patch -p0 < Embedding.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.dropout.Dropout' has changed. Saved a reverse patch to Dropout.patch. Run `patch -p0 < Dropout.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.rnn.LSTMCell' has changed. Saved a reverse patch to LSTMCell.patch. Run `patch -p0 < LSTMCell.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:657: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Sigmoid' has changed. Saved a reverse patch to Sigmoid.patch. Run `patch -p0 < Sigmoid.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)

here i set torch.nn.Module.dump_patches=True as per first error but both the time i am getting same output for every image.

output is :: a close up of a black and white background

can you help me through this error and please you specifi the version of pytorch, torch, torchvision that is compitable with current source code. i think upgraded version of these packages are remove or deprecated some functions.

EByrdS commented 4 years ago

I am having the same issue.

I am using Google Colab Pro to replicate this project, the provided checkpoint is stored in Google, so I only needed to add a direct access to it in my Drive.

I am using:

from google.colab import drive
drive.mount('/content/drive/', force_remount=True)

to access the files from the Colab.

Also, I had to do some changes in the imports, due to deprecations of imread and imresize, expecting the solution to have the exact same behavior (however, I do not think this caused the error):

# from scipy.misc import imread, imresize
from imageio import imread
from skimage.transform import resize as imresize

The cell that created the same output as the one who oppened this issue was (I used an image that already existed in the COCO14 dataset):

%run 'a-PyTorch-Tutorial-to-Image-Captioning/caption.py' --img='imgs_of_the_wild/coco_val_001.jpg' --model='files for A PyTorch Tutorial For Image Captioning/BEST_checkpoint_coco_5_cap_per_img_5_min_word_freq.pth.tar' --word_map='files for A PyTorch Tutorial For Image Captioning/WORDMAP_coco_5_cap_per_img_5_min_word_freq.json' --beam_size=3

It displayed all the above warnings, and the generated sentence for the image was: a close up of a black and white background, this output is generated for every image that I tried. image

I would prefer not to alter the code that triggered these warnings before ensuring that it works correctly, otherwise I wouldn't know if my changes broke the project or it was already broken. I started to train the model by myself, but it raised another error on the image COCO_val2014_000000320612.jpg, related to its shape:

img = imread(train_image_paths[9020]) # COCO_val2014_000000320612.jpg
img.shape # => (3, 375, 500)
img = imresize(img, (256, 256))
img = img.transpose(2, 0, 1)
img.shape # => (256, 256, 500)

assert img.shape == (3, 256, 256) # Raises AssertionError because (256, 256, 500) != (3, 256, 256)

I checked other images and their starting shape has the 3 at the end, like (640, 480, 3). I don't think that I generated this error on my project.

Has anyone replicated this project recently? I need to know if it works before doing any other changes that may or may not break it further.

Note: I will try solving the warnings and check if the checkpoint works correctly. But it may take some time.

shahhaard47 commented 4 years ago

Suggestion in this comment fixed the same caption generated issue for me. Comment out the following line:

# [line 38 in caption.py:caption_image_beam_search]
 img = imresize(img, (256, 256)) 

However, the source change warning still remains.

purveshpatel511 commented 4 years ago

@shahhaard47 Thanks.....It is solve the same caption issue.

shellyztian commented 4 years ago

I am having the same issue.

I am using Google Colab Pro to replicate this project, the provided checkpoint is stored in Google, so I only needed to add a direct access to it in my Drive.

I am using:

from google.colab import drive
drive.mount('/content/drive/', force_remount=True)

to access the files from the Colab.

Also, I had to do some changes in the imports, due to deprecations of imread and imresize, expecting the solution to have the exact same behavior (however, I do not think this caused the error):

# from scipy.misc import imread, imresize
from imageio import imread
from skimage.transform import resize as imresize

The cell that created the same output as the one who oppened this issue was (I used an image that already existed in the COCO14 dataset):

%run 'a-PyTorch-Tutorial-to-Image-Captioning/caption.py' --img='imgs_of_the_wild/coco_val_001.jpg' --model='files for A PyTorch Tutorial For Image Captioning/BEST_checkpoint_coco_5_cap_per_img_5_min_word_freq.pth.tar' --word_map='files for A PyTorch Tutorial For Image Captioning/WORDMAP_coco_5_cap_per_img_5_min_word_freq.json' --beam_size=3

It displayed all the above warnings, and the generated sentence for the image was: a close up of a black and white background, this output is generated for every image that I tried. image

I would prefer not to alter the code that triggered these warnings before ensuring that it works correctly, otherwise I wouldn't know if my changes broke the project or it was already broken. I started to train the model by myself, but it raised another error on the image COCO_val2014_000000320612.jpg, related to its shape:

img = imread(train_image_paths[9020]) # COCO_val2014_000000320612.jpg
img.shape # => (3, 375, 500)
img = imresize(img, (256, 256))
img = img.transpose(2, 0, 1)
img.shape # => (256, 256, 500)

assert img.shape == (3, 256, 256) # Raises AssertionError because (256, 256, 500) != (3, 256, 256)

I checked other images and their starting shape has the 3 at the end, like (640, 480, 3). I don't think that I generated this error on my project.

Has anyone replicated this project recently? I need to know if it works before doing any other changes that may or may not break it further.

Note: I will try solving the warnings and check if the checkpoint works correctly. But it may take some time.

I am having the same issue。Have you solved this problem?

EByrdS commented 4 years ago

Suggestion in this comment fixed the same caption generated issue for me. Comment out the following line:

# [line 38 in caption.py:caption_image_beam_search]
 img = imresize(img, (256, 256)) 

However, the source change warning still remains.

Yes, I commented this line and now the captions are genereted accurately. However, as the post indicates, the source code warning still remains.

ragvri commented 3 years ago

My source code warnings got removed when I used my CPU.