Using mergekit-moe command: ValueError: The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder` for them.

[nkotak]

Received this error below, Not sure how to proceed. Thank you!!

You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
INFO:root:Getting gate parameters...
Traceback (most recent call last):
  File "/opt/homebrew/bin/mergekit-moe", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/neal/AI/mergekit/mergekit/options.py", line 76, in wrapper
    f(*args, **kwargs)
  File "/Users/neal/AI/mergekit/mergekit/scripts/mixtral_moe.py", line 456, in main
    build(
  File "/Users/neal/AI/mergekit/mergekit/scripts/mixtral_moe.py", line 371, in build
    gate_vecs = get_gate_params(
                ^^^^^^^^^^^^^^^^
  File "/Users/neal/AI/mergekit/mergekit/scripts/mixtral_moe.py", line 157, in get_gate_params
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3850, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3979, in _load_pretrained_model
    raise ValueError(
ValueError: The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder` for them. Alternatively, make sure you have `safetensors` installed if the model you are using offers the weights in this format. ```

[nkotak]

I did find a thread about this again but I was wondering why this would occur since im running this on an Apple Silicon machine with 96gb of ram and the models are all 34B models. Could it be the number of models that is causing the issue?

[cg123]

Have you tried increasing the maximum amount of memory allocated to your GPU? I think by default OSX is set up to only allow you to use ~half of your RAM for the GPU. 34B will take at least 68GB plus some overhead for context length. I think you should be able to get it to work if you play with the setting but I can't say for sure as I don't have a comparable Mac to test with. Here's a post with some info on how: https://old.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_increase_vram_allocation_with_sudo_sysctl/