search

arcee-ai / mergekit

Tools for merging pretrained large language models.
GNU Lesser General Public License v3.0
4.57k stars 406 forks source link

Merge arbitrary pytorch models #335

Open cg123 opened 4 months ago

cg123 commented 4 months ago

Done in a dumb way for now. Going to revisit this and make it more general at some point in the future.

To try:

merge_method: dare_ties
base_model: model_base.safetensors
models:
  - model: model_a.safetensors
    parameters:
      density: 0.6
      weight: 0.7
  - model: model_b.safetensors
    parameters:
      density: 0.4
      weight: 0.3
parameters:
  normalize: true
  int8_mask: true

mergekit-pytorch ./config.yaml ./merged_model_folder [--tensor-intersection] [--tensor-union] [--cuda]