search

allenporter / home-assistant-datasets

This package is a collection of datasets for evaluating AI Models in the context of Home Assistant.
https://allenporter.github.io/home-assistant-datasets
22 stars 1 forks source link

Add results for qwen2.5 #44

Closed tannisroot closed 3 days ago

tannisroot commented 4 days ago

Using instructions from https://github.com/allenporter/home-assistant-datasets/issues/40#issuecomment-2372818481, I've evaluated qwen2.5 for assist and assist-mini benchmarks. The evaluation was done using local Ollama running on RX 7900 XTX with ROCM backend. Closes https://github.com/allenporter/home-assistant-datasets/issues/43

allenporter commented 3 days ago

This looks great. It looks like for some reason old reports are now being sorted in a different way than before. Would you mind reverting the reports that are for older versions? I'd like to preserve change history.

I'm surprised this is happening given the code should be dumping with sorted keys off.

allenporter commented 3 days ago

Lint error should be fixed in https://github.com/allenporter/home-assistant-datasets/commit/c9d1fa433a6c69d28ba03289f12dfc3dabacd667

tannisroot commented 3 days ago

@allenporter removed changes to old reports!

allenporter commented 3 days ago

Thank you @tannisroot 👍🏼