Open philschmid opened 2 months ago
Hey Philipp,
Thanks a lot for your nice commits!
To avoid breaking the original docs, I merged partial commits manually.
I acknowledged your help in the last of README, thanks again!
Hi @philschmid ,
Thank you for your PR, I was working on something similar ;) In inference, could it be that you missed usage of padding side as you are overwriting build tokenizer? When using local chat I have sometimes issues with empty responses which might be due to padding side unequal true. What do you think?
See in your Local Chat :)
regards and thank you for your efforts! Carsten
Hello,
I was playing around with mix_eval and noticed something, which i addressed in a fork to properly use it in my setup. First as #11 also opened there are some missing
__init__.py
which prevent remote installs withpip install git+https://github.com/Psycoy/MixEval --upgrade
. Additionally the dependencies are very hard which makes it harder to integrate in existing environments.I created a fork to make usage more easier.
This is a fork of the original MixEval repository. The original repository can be found here. I created this fork to make the integration and use of MixEval easier during the training of new models. This Fork includes several improved feature to make usages easier and more flexible. Including:
transformers
Getting started
_Note: If you want to evaluate models that are not included Take a look here. Zephyr example here._
Evaluation open LLMs
Remote Hugging Face model with existing config:
Using vLLM/TGI with hosted or local API:
start you environment
run the following command
Takes around 5 minutes to evaluate.
Local Hugging Face model from path:
Remote Hugging Face model without config and defaults
_Note: We use the model name
local_chat
to avoid the need for a config file and load it from the Hugging Face model hub._