search

allenporter / home-assistant-datasets

This package is a collection of datasets for evaluating AI Models in the context of Home Assistant.
https://allenporter.github.io/home-assistant-datasets
22 stars 1 forks source link

Add evaluation qwen2.5 LLM #43

Closed xiasi0 closed 3 days ago

xiasi0 commented 5 days ago

I performed very well in the Chinese environment (the test was not professional), and I hope you can add a complete evaluation of the English environment. thank you

tannisroot commented 4 days ago

I've performed my own evaluation here https://github.com/tannisroot/home-assistant-datasets/blob/qwen2.5/reports/README.md

xiasi0 commented 4 days ago

@tannisroot I was really surprised to see your evaluation results. I didn't expect it to perform well in an English environment. Just like Llama3.1, Chinese support is not user-friendly and almost unusable. This saved me a lot of OpenAI tokens. thank you！