search

arcee-ai / mergekit

Tools for merging pretrained large language models.
GNU Lesser General Public License v3.0
4.74k stars 434 forks source link

mergekit-evolve doesn't account for higher_is_better: false tasks. #377

Open mekaneeky opened 3 months ago

mekaneeky commented 3 months ago

In L317 in scripts/evolve.py a negative sign is added to make the minimization optimization in the cma library work for acc metrics. However, there is no option to reverse it for higher_is_better: false metrics. return [-x["score"] for x in res] # maximize This reflects in the optimization process and the best merge storage where the highest score model (the worst) is stored as best. A hacky solution I do at the moment is to just edit out the '-'. I am willing to write this nano feature. However, I am not familiar with the architecture/design. Is there a contribution style guide?

cg123 commented 2 months ago

I've just been using a negative weight for higher_is_better tasks. :) Contributions more than welcome, though - there's no written contribution guide at the moment but there is a pre-commit hook that enforces code style.