Flower dataset FID and IS reproduce

[nests01]

Hi, I loaded the pretrained model you provided and synthesized images. However, the FID is 241, and IS is 2.0. Meanwhile, the visualized results are very blurred. Could you please give me any advise about these results? Thanks very much!

[senmaoy]

1)Obviously, there is something wrong with your code or data because both the code and pre-trained model have been tested by different people. 2)It's usually troublesome to find a bug when python reports no error. I recommend you delete all the code and models and download them again.

[xouyang0079]

1)Obviously, there is something wrong with your code or data because both the code and pre-trained model have been tested by different people. 2)It's usually troublesome to find a bug when python reports no error. I recommend you delete all the code and models and download them again.

1) I tried on the CUB bird dataset with your pretrained model and can reach the results shown in the paper; I deleted all the models and codes and still got the same bad results. 2) May I ask whether you followed the previous work which split the flower dataset into 80 classes in training data and 20 classes (class 1 to 20) in testing data?

[senmaoy]

1)Can you train the model on flower dataset from scratch successfully? It will not take long to train a new flower model. If the training is normal, I will updata a new flower model. 2)Different from CUB and COCO, flower dataset uses all the training data for training. Becuase recent works ignored this dataset and it's not reasonable to totally seperate 20 classes from training data. I think this setting is better.

[xouyang0079]

1)Can you train the model on flower dataset from scratch successfully? It will not take long to train a new flower model. If the training is normal, I will updata a new flower model. 2)Different from CUB and COCO, flower dataset uses all the training data for training. Becuase recent works ignored this dataset and it's not reasonable to totally seperate 20 classes from training data. I think this setting is better.

There are 102 classes in total in flower dataset. Do you mean to use all 102 classes for training? In addition, could you please provide more details on how to set up the evaluation data? Thanks very much!

[senmaoy]

Using classes never appear in the training data for evaluation is strange. Like evaluation on imagenet, using all the training data for computing the FID score is more reasonable.

[xouyang0079]

Using classes never appear in the training data for evaluation is strange. Like evaluation on imagenet, using all the training data for computing the FID score is more reasonable.

In conclusion, you use all 102 classes for training in the flower dataset; to evaluate the IS and FID scores, you use 20 classes from testing data for 30k images synthesis, and 30k real images from all 102 classes, right?

[senmaoy]

I'm afraid that you mistook the above replies, there are only several thousand real images on the flower dataset. On the flower dataset, all the training data(102 classes) is used to train the model and compute FID. Thankyou for your paitience

[xouyang0079]

I'm afraid that you mistook the above replies, there are only several thousand real images on the flower dataset. On the flower dataset, all the training data(102 classes) is used to train the model and compute FID. Thankyou for your paitience

Thanks for your clarification. In the CUB bird dataset, to evaluate the IS and FID scores, you use testing data for 30k images synthesis, and 30k real images from the training dataset, right?

[senmaoy]

Hey, there are also only several thousand real images on the CUB dataset for training. The detailed test and training setting have been discussed in this issue(https://github.com/senmaoy/RAT-GAN/issues/3). On CUB, we follow other works and the setting is the same as others except for an inception model fine-tuned on CUB.

[xouyang0079]

1)Can you train the model on flower dataset from scratch successfully? It will not take long to train a new flower model. If the training is normal, I will updata a new flower model. 2)Different from CUB and COCO, flower dataset uses all the training data for training. Becuase recent works ignored this dataset and it's not reasonable to totally seperate 20 classes from training data. I think this setting is better.

Hi, I trained on the flower dataset for 600 epochs. The best IS is 3.45 and FID is 25.75. Could you please upload a new flower model? Thanks very much!

[senmaoy]

This result is terrible, i'm afraid that there is something wrong in your code. Anyway, I'll check the released flower model

[xouyang0079]

This result is terrible, i'm afraid that there is something wrong in your code. Anyway, I'll check the released flower model

I did nothing change to your code. Thanks for your checking.

[senmaoy]

I download the released flower model, and the result is ok. If your pytorch version or something else is different?

[senmaoy]

Do you use my pre-trained flower encoder to produce text embeddings?

[xouyang0079]

I use your pretrained text encoder and generator model. I tried on three different machines, they all generated very blurred and dim images as shown below:

[xouyang0079]

In main.py file, line 59: netG.load_state_dict(torch.load('../models/%s/netG_600.pth'%(cfg.CONFIG_NAME)))

I saw you loaded pretrained model in this path. Did you replace this model with the released model when you reproduced the results?

[senmaoy]

Yes, I download the /netG_600 from google drive. I guess that some layers or text feature is abnormal on your machine. Hence, both the pretrained model and training from scratch is wrong. Considering that your CUB results is OK, your text feature may be the cuase. I recomend to check the text encoder again. I suspect that you use the text encoder of cub or coco to encode the text on flower dataset. At the same time, I will also check the pre-trained text encoder

[senmaoy]

Hi, the pre-trained text encoder is also OK. If your still can't sovle the problem, welcome to join our wechat group and we can help you more directly.

[senmaoy]

I'm really sorry that I have used more affine blocks in the flower model carelessly. I only train the flower and coco only once and there are no other models left. Hence, to use the pre-trained model, please replace the original model.py with model_flower.py.