Phi 3 medium 128k bad generations

[steampunque]

b2985, 4070 full GPU offload.

Testing some quants (my own, no imatrix) with Phi 3 medium 128k and seeing problems in generation with Q4_K_S and Q4_0 and Q4_K_M and seeing different artifacts at all the levels. I tried generating files from f32, f16, and bf16 converts on Q4_K_S and they all gave identical results.

PROMPT: Name the top five physicists involved in the development of quantum mechanics.

Q4_K_M : quantum mechan artifact.

1. Max Planck: Max Planck is considered the father of quantum mechan by many. He introduced the concept of quantization of energy in 1900, which laid the foundation for quantum mechanics.

2. Albert Einstein: Albert Einstein made significant contributions to quantum mechanics, including his explanation of the photoelectric effect, for which he was awarded the Nobel Prize in Physics in 1921.

3. Niels Bohr: Niels Bohr developed the Bohr model of the atom, which incorporated quantum theory to explain the behavior of electrons in atoms. He also made significant contributions to the understanding of atomic structure and quantum mechanics.

4. Werner Heisenberg: Werner Heisenberg is known for his development of matrix mechanics, one of the first formulations of quantum mechanics. He also introduced the uncertainty principle, which states that it is impossible to simultaneously measure the position and momentum of a particle with absolute precision.

5. Erwin Schrödinger: Erwin Schrödinger developed wave mechanics, another formulation of quantum mechanics, and introduced the Schrödinger equation, which describes the behavior of quantum systems. He also made significant contributions to the understanding of atomic and molecular structure.

Q4_K_S: Generates an aborted output with artifacts.

1. Max Planck: Max Planck is considered the father of quantum mechanCURRENCY: USD

Q4_0: many artifacts (mechan, in 1)

The development of quantum mechan

The top five physicists involved in the development of quantum mechanics are:

1. Max Planck: He is considered the father of quantum theory. In 1

2. Niels Bohr: He developed the Bohr model of the atom, which explained the behavior of electrons in atoms.

3. Werner Heisenberg: He formulated the uncertainty principle, which states that the position and momentum of a particle cannot be simultaneously measured with arbitrary precision.

4. Erwin Schrödinger: He developed the Schrödinger equation, which describes the behavior of quantum systems.

5. Paul Dirac: He formulated the Dirac equation, which describes the behavior of relativistic electrons.

[ggerganov]

Without showing the commands that you are using we can't help you

[steampunque]

Without showing the commands that you are using we can't help you

convert: python3 convert-hf-to-gguf.py --outtype bf16 ./Phi-3-medium-128k-instruct quantize: quantize ./Phi-3-medium-128k-instruct/ggml-model-bf16.gguf Phi-3-medium-128k-instruct.Q4_K_S.gguf Q4_K_S run: main -m Phi-3-medium-128k-instruct.Q4_K_S.gguf --color -n -1 --multiline-input --interactive-first --log-disable -ngl 41 -c 8192 -b 128 -fa -n 8192 --keep 0 --temp 0.0 --dynatemp-range 0.0 --dynatemp-exp 1.0 --top-k 40 --top-p 0.95 --typical 1.0 --min-p 0.00 --repeat-last-n 64 --repeat-penalty 1.0 --presence-penalty 0.0 --frequency-penalty 0.0 --tfs 1.0 --mirostat 0 --mirostat-lr 0.1 --mirostat-ent 5.0 -p "" --in-prefix "<|user|>\n" --in-suffix "<|end|>\n<|assistant|>\n" prompt: Name the top five physicists involved in the development of quantum mechanics. Output with artifacts: <|end|>\n<|assistant|>\n 1. Max Planck: Max Planck is considered the father of quantum mechanCURRENCY: He introduced the concept of quantized energy levels in 1

Phi 3 mini works fine with both 4k and 128k versions, medium is bad with generation artifacts on both 4k and 128k versions.

[arnfaldur]

Please use code blocks as per these instructions.

It's very hard to read otherwise.

[foldl]

Such artifacts also occur in Q8. I believe this is caused by the model itself, although quantization may worsen it.