search

fangyuan-ksgk / Evolutionary-Model-Merge

Unofficial Implementation of Evolutionary Model Merging
Apache License 2.0
33 stars 2 forks source link

Question about your approach #1

Open JinSeoung-Oh opened 7 months ago

JinSeoung-Oh commented 7 months ago

Hi, at first, I want to say thank for your great work. I just some question about your approach. In your code, you used Mergekit for reproducing "Evolutionary Optimization of Model Merging Recipes" paper, right? Or it maybe, you just build your own method not reproduce the paper.

Anyway, the line number 146 in merge.py, you used "slerp". If my understanding is correct, "slerp" method merge just two model. So, if I want to merge 3 or 4 model, how can I do that..? I'm not familiar with Mergekit, so I don't know how.

And in the "Evolutionary Optimization of Model Merging Recipes" paper, author merged parameter space and data flow space of each LLM But I cannot figure out how can you marge data flow space.

Because I don't know much about Mergekit, my question may be naive. I hope it won't bother you. Thank you

wangyuchen333 commented 6 months ago

Slerp is for parameter space.