search

Lightning-AI / litgpt

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.
https://lightning.ai
Apache License 2.0
6.85k stars 726 forks source link

Avoid remote code warning in evaluation harness #1355

Closed awaelchli closed 3 weeks ago

awaelchli commented 3 weeks ago

Avoids the following warning:

/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/datasets/load.py:1461: FutureWarning: The repository for hails/mmlu_no_train contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hails/mmlu_no_train You can avoid this message in future by passing the argument trust_remote_code=True. Passing trust_remote_code=True will be mandatory to load this dataset from the next major release of datasets. warnings.warn(

For example when you run litgpt evaluate --tasks mmlu".

awaelchli commented 3 weeks ago

Sorry, it doesn't solve the problem because this warning is from the dataset not the model. Can't be addressed by us as far as I can tell.