search

Alpha-VLLM / LLaMA2-Accessory

An Open-source Toolkit for LLM Development
https://llama2-accessory.readthedocs.io/
Other
2.63k stars 168 forks source link

Number of parameters Sphinx #106

Closed ThierryDeruyttere closed 7 months ago

ThierryDeruyttere commented 8 months ago

Hi,

Thanks for the awesome Sphinx model! I was just wondering how many parameters this model has. I can't seem to find this in the report.

linziyi96 commented 8 months ago

SPHINX contains about 16.5B parameters, among which around 13B are in the language model and are trained, around 3.5B are in the visual encoders and are frozen (1B in QFormer, 0.85B in OpenCLIP ConvNext-xxl, 1B in DINOv2 ViT-g/14, 0.4B in CLIP ViT-L).