search

allenporter / home-assistant-datasets

This package is a collection of datasets for evaluating AI Models in the context of Home Assistant.
https://allenporter.github.io/home-assistant-datasets
33 stars 3 forks source link

Add results for qwen2.5 #44

Closed tannisroot closed 2 months ago

tannisroot commented 2 months ago

Using instructions from https://github.com/allenporter/home-assistant-datasets/issues/40#issuecomment-2372818481, I've evaluated qwen2.5 for assist and assist-mini benchmarks. The evaluation was done using local Ollama running on RX 7900 XTX with ROCM backend. Closes https://github.com/allenporter/home-assistant-datasets/issues/43

allenporter commented 2 months ago

This looks great. It looks like for some reason old reports are now being sorted in a different way than before. Would you mind reverting the reports that are for older versions? I'd like to preserve change history.

I'm surprised this is happening given the code should be dumping with sorted keys off.

allenporter commented 2 months ago

Lint error should be fixed in https://github.com/allenporter/home-assistant-datasets/commit/c9d1fa433a6c69d28ba03289f12dfc3dabacd667

tannisroot commented 2 months ago

@allenporter removed changes to old reports!

allenporter commented 2 months ago

Thank you @tannisroot 👍🏼