search

microsoft / unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
https://aka.ms/GeneralAI
MIT License
20.11k stars 2.55k forks source link

A question about vocabulary size #580

Closed peiyingxin closed 2 years ago

peiyingxin commented 2 years ago

i want to know why vocabulary size is set to |V| = 8192 ？

thank you~~

NielsRogge commented 2 years ago

Hi,

This is a choice that OpenAI made when creating DALL-E. From the paper:

We use a large vocabulary size of 8192 to mitigate the loss of information. (...) This reduces the context size of the transformer by a factor of 192 without a large degradation in visual quality.

peiyingxin commented 2 years ago

Ok thank you~ I will read this paper~