Trouble converting Phi-3-small

[dnhkng]

Hi Turbo,

I have a fresh pull of the dev branch, and I deleted the ~/.cache/torch_extensions folder.

I am trying to convert Phi-3-small using this command:

python convert.py \
    -i /media/dnhkng/Storage/models/Phi-3-small-8k-instruct \
    -o /media/dnhkng/Storage/tmp \
    -cf /media/dnhkng/Storage/models/Phi-3-small-8k-instruct-8.0bpw \
    -b 8.0

However, this throws this issue:

 -- Beginning new job
 !! Warning: Output directory is not empty: /media/dnhkng/Storage/tmp
 -- Input: /media/dnhkng/Storage/models/Phi-3-small-8k-instruct
 -- Output: /media/dnhkng/Storage/tmp
 -- Using default calibration dataset
 -- Target bits per weight: 8.0 (decoder), 6 (head)
 -- Max shard size: 8192 MB
 -- Full model will be compiled to: /media/dnhkng/Storage/models/Phi-3-small-8k-instruct-8.0bpw
 !! Warning, unknown architecture: Phi3SmallForCausalLM
 !! Loading as LlamaForCausalLM
Traceback (most recent call last):
  File "/home/dnhkng/Documents/LLM/exllamav2/convert.py", line 1, in <module>
    import exllamav2.conversion.convert_exl2
  File "/home/dnhkng/Documents/LLM/exllamav2/exllamav2/conversion/convert_exl2.py", line 187, in <module>
    config.prepare()
  File "/home/dnhkng/Documents/LLM/exllamav2/exllamav2/config.py", line 201, in prepare
    self.norm_eps = read(read_config, float, self.arch.norm_eps_key)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dnhkng/Documents/LLM/exllamav2/exllamav2/config.py", line 43, in read
    raise ValueError(f"Missing any of the following keys: {keys}")
ValueError: Missing any of the following keys: ['rms_norm_eps']

I saw that you already converted some of the other Phi-3 models, so any tips would be much appreciated!

[DocShotgun]

Phi 3 small is actually an entirely different architecture than mini and medium.

[remichu-ai]

Will there be plan to support it? I dont find anh phi 3 small exl guess that it is not supported currently. From latest open llm benchmark it actually is better than phi medium

[turboderp]

Phi3-small is indeed a completely different architecture. I simply don't have the time right now, and given all the other new architectures coming out almost every day it seems, I have to prioritize.