search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.8k stars 239 forks source link

How can I use HELM to evaluate local model? #2800

Open Zoey384 opened 2 weeks ago

Zoey384 commented 2 weeks ago

I modified the Huggingface's model stucture, and I wanted to use HELM to evaluate my model's performance. But I met some problems. I found I could create the files model_deployments.yaml, model_metadata.yaml and tokenizer_configs.yaml, but I cannot write my local model's path. How can I use my local model after I created the files model_deployments.yaml, model_metadata.yaml and tokenizer_configs.yaml in prod_env/ ? Must I upload my model to Huggingface?
Need I create a file like '/helm/src/helm/clients/huggingface_client.py'?

Many thanks!

yifanmai commented 2 weeks ago

Hi @Zoey384 - if your Hugging Face model checkpoint is on local disk, you can set pretrained_model_name_or_path: /path/to/your/checkpoint inside client_spec in your model_deployments.yaml as below. You do not need to upload your model to Hugging Face Hub.

https://github.com/stanford-crfm/helm/blob/fb9eb8941603c7b9d9dc5d421d94009dce711407/src/helm/config/model_deployments.yaml#L894-L897