search

InternLM / InternLM-XComposer

InternLM-XComposer2 is a groundbreaking vision-language large model (VLLM) excelling in free-form text-image composition and comprehension.
1.92k stars 121 forks source link

Context length during training? #247

Closed ys-zong closed 2 months ago

ys-zong commented 2 months ago

Hi, thanks for the nice work! I saw the max_position_embeddings is set to 32768 in the config.json. This is quite large. I wonder what is the context length you use during training? I didn't find it in the report. Many thanks!

LightDXY commented 2 months ago

hi, for the xcomposer2, most data has less than 4096 tokens. 32768 is the max length supported by the LLM backbone InternLM2. For our new xcomposer2-4khd, it is trained with nearly 8k tokens per sample.