AttnGAN evaluation error

[rracinskij]

Hi,

I get an error while trying to generate images using the pretrained AttnGAN model: sh sample.sh coco-attngan ...

  File "main.py", line 86, in gen_example
    algo.gen_example(data_dic)
  File ".../multiple-objects-gan-master/code/coco/attngan/trainer.py", line 604, in gen_example
    netG.load_state_dict(state_dict)
RuntimeError: Error(s) in loading state_dict for G_NET:
Unexpected key(s) in state_dict: "netG".

[tohinz]

Can you give me a print out of the state_dict? For a quick fix try netG.load_state_dict(state_dict["netG"]) to see if that works.

[rracinskij]

Thank you, the quick fix solved the problem, as the state_dict contains a netG key only. So the line 604 of the code/coco/attngan/trainer.py should probably contain netG.load_state_dict(state_dict["netG"])

[tohinz]

Thanks for spotting this, I have just fixed this in the code.

[rracinskij]

Sorry for reopening - one more issue with sh sample.sh coco-attngan:

File "main.py", line 160, in <module>
    gen_example(dataset.wordtoix, algo)  # generate images for customized captions
  File "main.py", line 86, in gen_example
    algo.gen_example(data_dic)
  File "/home/.../multiple-objects-gan-master/code/coco/attngan/trainer.py", line 636, in gen_example
    fake_imgs, attention_maps, _, _ = netG(noise, sent_emb, words_embs, mask)
  File "/.../lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() takes exactly 7 arguments (5 given)

[tohinz]

Hi, the method you're trying to call is from the original AttnGAN implementation and is used to generate images from "novel" sentences (i.e. not from the validation set). To avoid this make sure you have B_VALIDATION: True in the cfg_eval.yml file.

If you want to use novel sentences to generate images you'll need to provide the two values for transf_matrices_inv and label_one_hot for each caption and call fake_imgs, attention_maps, _, _ = netG(noise, sent_emb, words_embs, mask, transf_matrices_inv, label_one_hot) Essentially you need to decide where in the image (with transf_matrices_inv) and what kind of objects (with label_one_hot) you want to generate. See datasets.py lines 331-339 to see how to obtain the transformation matrices from a scaled bounding box (bbox values are [x-coordinate, y-coordinate, width, height] each value scaled so it is between 0 and 1, e.g. [0.0, 0.0, 0.5, 0.5] for an object that covers the top left quarter of the image).

The labels are: 0 person 1 bicycle 2 car 3 motorcycle 4 airplane 5 bus 6 train 7 truck 8 boat 9 traffic light 10 fire hydrant 11 stop sign 12 parking meter 13 bench 14 bird 15 cat 16 dog 17 horse 18 sheep 19 cow 20 elephant 21 bear 22 zebra 23 giraffe 24 backpack 25 umbrella 26 handbag 27 tie 28 suitcase 29 frisbee 30 skis 31 snowboard 32 sports ball 33 kite 34 baseball bat 35 baseball glove 36 skateboard 37 surfboard 38 tennis racket 39 bottle 40 wine glass 41 cup 42 fork 43 knife 44 spoon 45 bowl 46 banana 47 apple 48 sandwich 49 orange 50 broccoli 51 carrot 52 hot dog 53 pizza 54 donut 55 cake 56 chair 57 couch 58 potted plant 59 bed 60 dining table 61 toilet 62 tv 63 laptop 64 mouse 65 remote 66 keyboard 67 cell phone 68 microwave 69 oven 70 toaster 71 sink 72 refrigerator 73 book 74 clock 75 vase 76 scissors 77 teddy bear 78 hair drier 79 toothbrush

[rracinskij]

Thank you for a detailed explanation. Could you please specify the correct dimensions of bbox, transf_matrices_inv and label_one_hot?

[tohinz]

The dimensions should be the following (num objects = how many objects you want to specify for the object pathway, in our case usually 3): bbox: [batch size, num objects, 4] trans_matrices_inv and transf_matrices: [batch size, num objects, 2, 3] label_one_hot: [batch size, num objects, 81]