save_pretrained_merged ruins my model

[Romiroz]

Good day        
        After load saved lora model, i save it to merged. And after load it from merged, i have generation like '+++++ 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False.
torch.cuda.empty_cache()
# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
fourbit_models = [
    "unsloth/Meta-Llama-3.1-8B-bnb-4bit",      # Llama-3.1 15 trillion tokens model 2x faster!
    "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    "unsloth/Meta-Llama-3.1-70B-bnb-4bit",
    "unsloth/Meta-Llama-3.1-405B-bnb-4bit",    # We also uploaded 4bit for 405b!
    "unsloth/Mistral-Nemo-Base-2407-bnb-4bit", # New Mistral 12b 2x faster!
    "unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit",
    "unsloth/mistral-7b-v0.3-bnb-4bit",        # Mistral v3 2x faster!
    "unsloth/mistral-7b-instruct-v0.3-bnb-4bit",
    "unsloth/Phi-3.5-mini-instruct",           # Phi-3.5 2x faster!
    "unsloth/Phi-3-medium-4k-instruct",
    "unsloth/gemma-2-9b-bnb-4bit",
    "unsloth/gemma-2-27b-bnb-4bit",            # Gemma 2x faster!
] # More models at https://huggingface.co/unsloth

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "nemo5/checkpoint-1260",
    max_seq_length = max_seq_length,
    dtype = torch.bfloat16,
    load_in_4bit = True,
    # device_map =  "cuda",
    #local_files_only = True,
    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)

model.save_pretrained_merged("merged_nemo51", tokenizer, save_method="merged_16bit") #,safe_serialization = False

[danielhanchen]

@Romiroz Apologies on the issue - it's best to ask on our Discord server for faster and async help

[mjschock]

@Romiroz I'm wondering if you might have found help through Discord or otherwise? I did notice that save_pretrained_merged seems to keep the tokenizer's padding side set to 'left' (you can check the tokenizer_config.json for this) and wonder if this might be the reason the generations are looking off.