Possibility to integrate multiple model types?

Leeroo-AI / mergoo

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

https://www.leeroo.com/

GNU Lesser General Public License v3.0

397 stars 25 forks source link

Possibility to integrate multiple model types? #16

Closed sammcj closed 4 months ago

sammcj commented 4 months ago

It would be neat if Mergoo could integrate multiple model types.

For example a great MoM for technical tasks might be something like:

Llama 3 8b
Phi 3 Medium 128k
CodeQwen1.5 7b
Nxcode-CQ-7B-orpo

Is this technically possible with Mergoo?

alirezamshi commented 4 months ago

Hi,

For the integration with mergoo, fine-tuned experts must have the same architecture. For model-wise integration, please check our router library (https://github.com/Leeroo-AI/leeroo_orchestrator).

sammcj commented 4 months ago

That's really helpful, thank you for your response Alireza. 😄