search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.89k stars 244 forks source link

Add Test for Optimum Intel #2674

Closed NoushNabi closed 4 months ago

NoushNabi commented 4 months ago

Added test for Optimum Intel.

yifanmai commented 4 months ago

Hi @NoushNabi, could you move this test into its own new job under jobs? It is not a unit test and should not be grouped with that.

You can do this by duplicating and modifying the Run HELM with minimal dependencies only test - in particular, you will have to run python3 -m pip install dist/crfm_helm-*.whl[openvino] instead of python3 -m pip install dist/crfm_helm-*.whl.

NoushNabi commented 4 months ago

Hi @yifanmai python3 -m pip install dist/crfm_helm-*.whl[openvino] was giving error since building was only producing crfm_helm-0.5.1-py3-none-any.whl. I added python3 -m pip install optimum[openvino] and it works on my machine but on Github Actions exits with error as Error: The operation was canceled. Can you please advise?

yifanmai commented 4 months ago

I think the following command should work i.e. resolve the wildcard first before passing the file name to pip:

python3 -m pip install "$(ls dist/crfm_helm-*.whl)[openvino]"
NoushNabi commented 4 months ago

@yifanmai Thanks for your reply. I appreciate it! Finally all tests for Optimum Intel pass successfully.

yifanmai commented 4 months ago

Lint is failing with:

src/helm/clients/huggingface_client.py:77:17: F401 'pathlib.Path' imported but unused

Could you please fix this?

NoushNabi commented 4 months ago

@yifanmai Sorry that I missed it. Done!