lucidrains / DALLE-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
MIT License
5.56k stars 643 forks source link

Bad result with vqgan #418

Open shizhediao opened 2 years ago

shizhediao commented 2 years ago

Hi, I am using VQGAN on the MSCOCO training dataset (also tried adding Visual Genome to construct a 1 Million dataset), but got a bad result. The pixels are wired.

image image image image

Here are my settings, thanks!

` transformer_dim = 512 rotary_emb = False image_fmap_size = 32 self.transformer = Transformer( dim = transformer_dim, causal = True, seq_len = seq_len, depth = self.config_visual_decoder.num_hidden_layers, heads = 8, dim_head = 64, reversible = False, attn_dropout = 0.0, ff_dropout = 0.0, attn_types = 'full', image_fmap_size = image_fmap_size, sparse_attn = False, stable = False, sandwich_norm = False, shift_tokens = False, rotary_emb = rotary_emb,

shared_attn_ids = None,

        # shared_ff_ids = None,
        # optimize_for_inference = False,
    )`

I looked up several previous issues and reports and notice that people usually get loss < 4.5 while my loss is around 5.4.

I use a large batch size (more than 3000) while others use a far smaller batch size (like 16), does that matter? Thanks

rom1504 commented 2 years ago

yes if you can use such a large batch size, it either means that you have hundred of gpus or that your model is too small you should increase the depth parameter, and probably train for longer. How long did you train?

shizhediao commented 2 years ago

Thanks for your quick reply! I have tried three settings, all of them are based on a 6layer Transformer.

  1. MSCOCO training data: 400,000 image-text-pairs
  2. MSCOCO + VG + some private data: around 6M image-text-pairs
  3. MSCOCO + VG + CC15M: around 16M image-text-pairs

To speed up the training, I use many A100 GPUs with only 15 epochs. The first setting could be finished in several hours.

The loss is from 7.0 -> 6.5 -> 5.4, and stuck at 5.4 from epoch5.

shizhediao commented 2 years ago

What could I do if I want to both speed up with large batch size and use only 6layer Transformer? Maybe larger learning rate and train longer? It seems that even with lr=1e-3, the loss will get stuck at 5.0

rom1504 commented 2 years ago

Try the default lr and depth 16 What distributor do you use ? Deepspeed? Horovod?

Increasing depth usually gets much better results

shizhediao commented 2 years ago

Try the default lr and depth 16 What distributor do you use ? Deepspeed? Horovod?

Increasing depth usually gets much better results

Neither, I am using NCCL constructed by a company. I suddenly found there is an issue said ""adamw" optimizer + weight decay = poor generations" https://github.com/lucidrains/DALLE-pytorch/issues/170 I am using adamw + weight decay0.01, does that matter?