search

huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
https://huggingface.co/docs/evaluate
Apache License 2.0
1.9k stars 235 forks source link

Allow for specify coda device in perplexity evaluation #568

Open manuelbrack opened 3 months ago

manuelbrack commented 3 months ago

The current implementation of perplexity.compute does not allow to provide a specific coda device (e.g. cuda:1 instead of just cuda.

https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/metrics/perplexity/perplexity.py#L108

Should probably be changed to something along the lines of: 

assert device in ["gpu", "cpu", "cuda"] or re.search(r'^cuda:[0-9]+$', device), "device should be either gpu or cpu."