search

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
845 stars 100 forks source link

Custom task execution #352

Closed TomasJavurek closed 1 month ago

TomasJavurek commented 1 month ago

I've implemented a custom task and also changed BaseModelConfig argument: trust_remote_code: bool = True

but when executing: lighteval accelerate --model_args "pretrained=gpt2" --tasks "community|{custom_task}" --custom_tasks "community_tasks/kinit_evals.py" --output_dir "./evals"

I got: File "C:\Users\tomas.javurek\AppData\Local\Programs\Python\Python312\Lib\site-packages\datasets\load.py", line 131, in resolve_trust_remote_coe raise ValueError( ValueError: The repository for kinit_evals contains custom code which must be executed to correctly load the dataset. You can inspect the reposi. Please pass the argument trust_remote_code=True to allow custom code to be run.

I am not sure where to pass it.

clefourrier commented 1 month ago

Hi! It's in your custom task definition, not for the model. Here the error is when loading the data from your task. You'll also need to make sure that you're using the latest version of lighteval.

Note: please follow the issue template next time (provide the full stack trace, full command, custom code, etc) so we have an easier time debugging!