search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.77k stars 234 forks source link

Support for Hugging Face Local Models #2690

Open NoushNabi opened 1 month ago

NoushNabi commented 1 month ago

One question regarding support for Hugging Face local model, I found this:

https://github.com/stanford-crfm/helm/blob/7ead1d6e41e91683ff6b06a94bd9e2ed7b629d0a/docs/huggingface_models.md?plain=1#L13

saying "Models at local file paths are not supported." Also, I have found this link: https://github.com/stanford-crfm/helm/pull/1505, which I am not able to find mentioned files. Could you please confirm the first link is the most updated and accurate one? Many thanks!

yifanmai commented 1 month ago

The docs are outdated - we do support Hugging Face models at local file paths now.