search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.89k stars 244 forks source link

Added PaliGemma #2683

Closed JosselinSomervilleRoberts closed 4 months ago

JosselinSomervilleRoberts commented 4 months ago

Not sure how to deal with the dependency issue. PaliGemma needs transformers 4.41 so I just installed it even with the conflicts and it ran fine.

teetone commented 4 months ago

I think we're good to merge once the checks pass.

teetone commented 4 months ago

Not sure how to deal with the dependency issue. PaliGemma needs transformers 4.41 so I just installed it even with the conflicts and it ran fine.

Following up in a separate issue, since the code for the models is self-contained.