search

mikeybellissimo / LoRA-MPT

A repo for finetuning MPT using LoRA. It is currently configured to work with the Alpaca dataset from Stanford but can easily be adapted to use another.
Apache License 2.0
18 stars 7 forks source link

Evaluation #9

Open benam2 opened 1 year ago

benam2 commented 1 year ago

Hi again, I have fined tuned the Instruct model on a new dataset and now i want to evalaute the performance of the model. I followed the instruction and https://github.com/mikeybellissimo/lm-evaluation-harness successfully passed.

However when I run the script %sh python src/eval.py --model mpt-causal --model_args pretrained=mosaicml/mpt-7b-instruct,trust_remote_code=True,load_in_8bit=True,peft=lora-mpt --tasks hellaswag it raises

`Traceback (most recent call last): File "/Workspace/Repos/LoRA-MPT/src/eval.py", line 8, in from lm_eval import tasks, evaluator, utils ModuleNotFoundError: No module named 'lm_eval'

Just as a note when I get list of the libraries with pip freeze, it gives me this # Editable install with no version control (lm-eval==0.3.0) -e /Workspace/Repos/lm-evaluation-harness

Just putting the last line of the successful installation in case it helps: Installing collected packages: lm-eval Attempting uninstall: lm-eval Found existing installation: lm_eval 0.3.0 Uninstalling lm_eval-0.3.0: Successfully uninstalled lm_eval-0.3.0 Running setup.py develop for lm-eval Successfully installed lm-eval

Appreciate any input on how to fix this.

benam2 commented 1 year ago

@mikeybellissimo could you please have a look at this issue? I guess no one may have used the evaluation?

mikeybellissimo commented 1 year ago

Hi, did you clone my fork of lm-eval from the GitHub link in the read me and then pip -e install . From within that directory?

benam2 commented 1 year ago

Thanks for the quick reply @mikeybellissimo . Yea, I cloned that repo and then pip install . and then ran the python src/eval.py --model mpt-causal --model_args pretrained=mosaicml/mpt-7b-instruct,trust_remote_code=True,load_in_8bit=True,peft=lora-mpt --tasks hellaswag but it kept raisisn the rror mentioned above.

Traceback (most recent call last): File "/Workspace/Repos/LoRA-MPT/src/eval.py", line 8, in from lm_eval import tasks, evaluator, utils ModuleNotFoundError: No module named 'lm_eval

And this is the last line of the successful installation: Installing collected packages: lm-eval Attempting uninstall: lm-eval Found existing installation: lm_eval 0.3.0 Uninstalling lm_eval-0.3.0: Successfully uninstalled lm_eval-0.3.0 Running setup.py develop for lm-eval Successfully installed lm-eval

As you see in the installation it is lm-eval (UPPERCASE) but in the script for importing the library is lm_eval (UNDERSCORE)