RuntimeError: Unsloth: Quantization failed

[kolomichyk]

Hi! I'm not using Kaggle, but how can I get that exception? I'm just trying to save gguf model.

model.save_pretrained_gguf("model", tokenizer, quantization_method = "f16")

RuntimeError: Unsloth: Quantization failed for ./model/unsloth.F16.gguf You are in a Kaggle environment, which might be the reason this is failing. Kaggle only provides 20GB of disk space. Merging to 16bit for 7b models use 16GB of space. This means using model.{save_pretrained/push_to_hub}_merged works, but `model.{save_pretrained/push_to_hub}_gguf will use too much disk space. I suggest you to save the 16bit model first, then use manual llama.cpp conversion.

[danielhanchen]

Oh weird my environment variable checking says you're in a Kaggle env - but anyways it just means you don't have enough disk space

[danielhanchen]

Best to save the LoRA adapters, and convert on a larger disk space

[kolomichyk]

Is there any way I can get around this Kaggle environment variable?

вт, 30 июл. 2024 г. в 20:07, Daniel Han @.***>:

Best to save the LoRA adapters, and convert on a larger disk space

— Reply to this email directly, view it on GitHub https://github.com/unslothai/unsloth/issues/835#issuecomment-2258818628, or unsubscribe https://github.com/notifications/unsubscribe-auth/APIQY67NG7RK33RFLRDIO6DZO7B47AVCNFSM6AAAAABLWAAGL2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJYHAYTQNRSHA . You are receiving this because you authored the thread.Message ID: @.***>

[danielhanchen]

It's not the Kaggle issue - it just detects you do not have enough disk space to save the 16bit model. But weirdly it also means you have some kaggle keys in the environment vars ie IS_KAGGLE_ENVIRONMENT = "KAGGLE_" in keynames