search

AILab-CVC / SEED

Official implementation of SEED-LLaMA (ICLR 2024).
https://ailab-cvc.github.io/seed
Other
515 stars 30 forks source link

Hyperparameter for training SEED Tokenizer #16

Open Cheolhyun-Mun opened 6 months ago

Cheolhyun-Mun commented 6 months ago

Hi! Thank you for the wonderful work.

I wonder if you can provide detailed information on training SEED Tokenizer. I cannot find the hyperparameter for training SEED Tokenizer in your paper.

Also, I have another question. In the paper, SEED Tokenizer training is divided into two stages. Does that mean the Q-former is pre-trained in stage 1 and then the Q-former, codebook, decoder, and MLP are trained in stage 2?

Thank you.

geyuying commented 4 months ago

We have released the training code of SEED-LLaMa, including SEED tokenizer, Multimodal LLM pretraining and instruction tuning. Our Multimodal LLM training codebase supports 1. large-scale multi-node training with deepspeed 2. highly-efficient multiple training datapipes.