best fid score?

[yangsenwxy]

Hi, Thank you so much for your work How to choose the weight of the best FID score, I used your code to reproduce, none of them reached the performance to the original paper, My model in the FlintstonesSV of FID is 30

[xichenpan]

@yangsenwxy Hi, could you please provide some visual samples? and also the training config like how many epochs, freeze clip/blip/resnet or not.

[xichenpan]

btw, in our experiment, we just use the ckpt in 50 epochs, cause the loss is irrelevant with the fid score.

[yangsenwxy]

can I have you wechat?I tell you by email? ys810137152@gmail.com

[yangsenwxy]

This is my config

[xichenpan]

can I have you wechat?I tell you by email? ys810137152@gmail.com

@yangsenwxy Yeah, I am happy to chat. But I prefer to discuss in this thread because this is a new implementation and we haven't train a model using this repo. If we can figure out the problem, it may help a lot~

[xichenpan]

This is my config

It seems the init_lr is too large, you may try our config of 1e-5. Also, could you provide your sample config, like scheduler and steps?

[yangsenwxy]

I will give you some exsample

[xichenpan]

I will give you some exsample

You may follow the config given in https://github.com/Flash-321/ARLDM/blob/main/config.yaml (which is the config for the report performance.) The ddim scheduler with 50 steps provide a 2-3 higher FID score result. We suggest using ddim-6-250 or pndm-7.5-50 (this is faster and provide a 1-2 higher FID score)

[xichenpan]

I will give you some exsample

Thanks a lot! Only the generated images is enough (caption is not related to FID score)

[yangsenwxy]

链接：https://pan.baidu.com/s/1s4WU9xdS7Qn_XO-O2kvwaA 提取码：n1dl --来自百度网盘超级会员V5的分享

[yangsenwxy]

here

[xichenpan]

here

Copy! So I think it is a training issue, since the generated visual stories are unacceptable.

[xichenpan]

@yangsenwxy Have you ever try to train the model for 50 epochs using a init_lr of 1e-5?

[yangsenwxy]

I will try it

[xichenpan]

I will try it

@yangsenwxy Thanks! I have tried using a large lr, but it produce a bad performance when training for many epochs. You can also try to train the model for 5 epochs using a lr of 1e-4 and see if it works. I tried this setting in my early experiment and it can also generate reasonable results. Feel free to at me in this thread if there is an update or further issue!

[yangsenwxy]

Ok, let me try, do you mean smaller learning rate, smaller FID?

[xichenpan]

Ok, let me try, do you mean smaller learning rate, smaller FID?

@yangsenwxy Not actually, but large learning rate do not work well when you train the model really long.

[yangsenwxy]

@Flash-321 I used your exact same settings for the last 50 epoch and ran out a FID score of 25

[xichenpan]

@yangsenwxy yeah, and do you also use the default setting during sampling?

[xichenpan]

also, may I see your learning rate curve to find out if the scheduler work correctly?

[yangsenwxy]

OK，wait for me for half an hour

[xichenpan]

Sure!

[xichenpan]

@yangsenwxy Really sorry about a bug in our code. I just check our dataset implementation, and found that during immigranting to Pytorch Lightning code, we forget to add the normalization for training data. Here is our internal implementation: https://github.com/Flash-321/ARLDM/blob/a24e2e94332eb86fcc071abb83aaf341006aa622/ARLDM.py#L47-L52 Here is the pytorch lightning implementation in current branch: https://github.com/Flash-321/ARLDM/blob/eb907e3717ac20f82dfba8e67fd55d95127de098/datasets/flintstones.py#L26-L31 This will do harm to the PTMs since Stable Diffusion is trained on these normalized images.

My sincere apologize for this issue. The FID score should be further improved after fixing this bug.

[xichenpan]

@yangsenwxy Really sorry about a bug in our code. I just check our dataset implementation, and found that during immigranting to Pytorch Lightning code, we forget to add the normalization for training data. Here is our internal implementation:

https://github.com/Flash-321/ARLDM/blob/a24e2e94332eb86fcc071abb83aaf341006aa622/ARLDM.py#L47-L52

Here is the pytorch lightning implementation in current branch: https://github.com/Flash-321/ARLDM/blob/eb907e3717ac20f82dfba8e67fd55d95127de098/datasets/flintstones.py#L26-L31

This will do harm to the PTMs since Stable Diffusion is trained on these normalized images. My sincere apologize for this issue. The FID score should be further improved after fixing this bug.

And this bug only happends in FlintstonesSV dataset, if you have done experiments in other datasets, the performance was not affected.

[yangsenwxy]

I discovered this bug a long time ago, and it has been corrected during the original training

[yangsenwxy]

@yangsenwxy yeah, and do you also use the default setting during sampling?

Yes

[yangsenwxy]

also, may I see your learning rate curve to find out if the scheduler work correctly?

[xichenpan]

@yangsenwxy

also, may I see your learning rate curve to find out if the scheduler work correctly?

Thanks, it looks correct.

[yangsenwxy]

Now,FID is 25.069521856381982

[xichenpan]

Now,FID is 25.069521856381982

Got it! Could you please post 1-2 visual samples on this thread? There are also several differences between the two implementation. Like in this version, we use stable diffusion v1.5 instead of v1.4, but I think it doesn't matter. We also use a gradient clip in our training process, I guess this may be a factor https://github.com/Flash-321/ARLDM/blob/a24e2e94332eb86fcc071abb83aaf341006aa622/config/config_flintstones.json#L51-L66 maybe you can go through the training log and see if the gradient is too large in some step? If so, I guess adding this config may help:

trainer = Trainer(gradient_clip_val=1.0)

You can also check our original internal implementation (https://github.com/Flash-321/ARLDM/tree/a24e2e94332eb86fcc071abb83aaf341006aa622) to find some difference, we can discuss them in this thread.

[yangsenwxy]

Here

[yangsenwxy]

[yangsenwxy]

Here ,some visual samples

[xichenpan]

@yangsenwxy Copy, your experiments help a lot, thanks for that! It looks like a training issue, the val loss should be around 0.11, and the visual samples are not reasonable and coherent. I recommend add clip_norm and train for 5 epochs in a learing rate of 1e-4 (this can help reduce traning cost.), so that we can find out whether it is due to the large gradient.

[yangsenwxy]

OK

[yangsenwxy]

@Flash-321 I have added clip_norm as you suggested, but the FID is 24. 24.87433417204761

[yangsenwxy]

loss is here

[yangsenwxy]

[xichenpan]

@yangsenwxy Hi, thanks for your feedback, it seems the conditioning part is ruined, since each frame is not coherent. I will check the implementation soon and also ask my mentor to release the ckpt and training log for reference.

[KyonP]

hello. I also have a difficult time reproducing your best-performing results on Pororo.

Can you share some config settings?

[xichenpan]

@KyonP Hi, our config is the same as https://github.com/xichenpan/ARLDM/blob/main/config.yaml However, as discussed in this thread, it is quite weird the implementation in Pytorch Lighting cannot fully reproduce the result. I spend a few hours finding the difference between the two version. However, I have mad too much modification so that I cannot distinguish what cause the problem. You are welcome to refer to our original implementation https://github.com/xichenpan/ARLDM/tree/a24e2e94332eb86fcc071abb83aaf341006aa622, if you find some bug in our pytorch lightning implementation, I am more than happy to know! Besides, we plan to release the trained weight so that you can use it to inference.

[KyonP]

@xichenpan I managed to achieve an FID score of around 18 with PL version of your code. It looks good enough for me right now 👍

I am looking forward to having your pre-trained weights.

BTW, is there any chance you will improve the inference speed of this code? it is very time-consuming.

[xichenpan]

@KyonP Great! I just asked my mentor, and he told me the release request have been approved by Alibaba, we will provide the pre trained weight this week!

[yxding95]

Hey, I found the same issue. @xichenpan Did you try to save the images first and then evaluate the FID? Or you only use the FID calculation in the "main.py" code? Cause I found the Inception Network in the code seems to lose ".eval()", which would norm the features and get a lower FID.

[TimandXiyu]

@xichenpan Hi, I am having a hard time finding the released weight, and I am trying to do some follow-ups of your work. If you haven't upload them yet, can you share them via google drive or maybe other platforms?

[kirbu123]

@xichenpan, Are your checkpoints ready ? If possible, will you be ready to share them. Can you explain, how to learn ARLDM with one available CUDA index. How to beat CUDA out of memory error using CLIP, BLIP, RESNET freezing or other methodics.

[kirbu123]

Can somebody (who made training) share checkpoints, because I have CUDA out of memory, training the model.