search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.77k stars 235 forks source link

Difference between HEIM and VHELM? #2726

Closed zhimin-z closed 4 weeks ago

yifanmai commented 4 weeks ago

HEIM is for text-to-image models and VHELM is for vision-language models.

zhimin-z commented 4 weeks ago

HEIM is for text-to-image models and VHELM is for vision-language models.

Thanks for your answer. The vision-language model can do both text-to-image and image-to-text tasks, right?

yifanmai commented 3 weeks ago

No - of the current models on listed VHELM v1.0.0, only GPT-4o supports image generation. Additionally, GPT-4o's image generation is not generally available on OpenAI's API yet.