Is the generation greedy-decoding?

[w32zhong]

Hi, thanks for sharing the code.

It looks like the generation is in greedy mode by looking at the jacobi_forward(), however, I observe different outputs every time, and sometimes the outputs are just gibberish. I am using RTX 3060 with 8-bit quantization, not sure if the quantization is the cause of the issue here. If anyone has similar observations with full-precision using more advanced GPUs, please let me know.

Also, when outputs are gibberish, the speedup are really high presumably because there are a lot of repeating patterns. When the output is good, the speedup will become lower.

Here I attach some outputs examples:

Good one:

USER: tell me a few interesting facts about the sun and the moon.
ASSISTANT: 1. The sun is approximately 4.6 million miles away from Earth,while the moon is about 238,900 miles away.
2. The sun is about 864,000 milesin diameter, while the moon is only about 2,159 milesin diameter.
3. The sun is made up of hydrogen and helium, while the moon is primarily composed of silicon and iron.
4. The sun is a spherical object, while the moon has a slightly irregularshape due to its formation process.
5. The sun takes approximately 24 days to complete one rotation, while the moon takes approximately 27.3 days to complete one rotation around Earth.
6. The sun is approximately4.6 billion years old, while the moon is estimated to be around4.5 billion years old.
7. The sun's surfacetemperature is approximately 5,500°F (3,000°C), while the moon's surface temperature ranges from -273°F (-170°C) during the night to 260°F (127°C) during the day.
8. The sun is a star, while the moon is a natural satellite ofEarth.
9. The sun is the source of all energy on Earth,while the moon has no intrinsic energy source.
10. The sunis the largest object in the solar system, while the moon is the largest naturalsatellite in the solar system.
======================================SUMMARY=======================================================
Generated tokens:  336 Time:  30.96 s Throughput:  10.85 tokens/s Fast forwarding:  1.54 tokens/step
====================================================================================================

Repeating at the end:

USER: tell me a few interesting facts about the sun and the moon.
ASSISTANT: 1. The sun is a gigantic ball of plasma, with a diameterof approximately 1,392,600 kilometers (865,770 miles). This is over 100 timesthe diameter of Earth.
2. The sun is approximately 4.6billion years old, making it one of the oldest stars in the Milky Waygalaxy.
3. The sun's surface, or photosphere,has a temperature of approximately 5,500°C (9,932°F). This is hot enough to melt iron.
4. The sun's core, the innermost region, has atemperature of approximately 15,000,000°C(27,000,000°F). This ishot enough to cause nuclear fusion, the process that powers the sun.
5. The sun's gravity is so strong that it can hold together its ownmass, which is approximately 2×10³⁰ kilograms (4.5×10²⁶ pounds).
6. The sun's light takes approximately 8 minutes and 20 seconds to reach Earth.
7. The sun is a nearly perfect sphere, with a slight flattening at the poles due to the rotation of the sun.
8.The sun's rotation rate is approximately 25 days, which is muchslower than its convection rate, which is approximately 30 kilometersper second (18.6 miles per second).
9. The sun's magnetic field is approximately 100 times stronger than Earth'smagnetic field.
10. The sun's atmosphere, or corona, is approximately 1,000,000 kilometers (621,371 miles) wide, making it the largest structurein the solar system.

1. The moon is a natural satellite ofEarth, with a diameter of approximately 3,476 kilometers (2,160 miles).
2. The moon's surface iscovered in craters, created by impacts from asteroids and other celestial bodies over billions of years.
3. The moon's gravityis approximately 17% of Earth's gravity, which means that aperson on the moon would weigh approximately 17% of what they weigh on Earth.
4. The moon's rotation rate is approximately 27.32166666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666
======================================SUMMARY=======================================================
Generated tokens:  1040 Time:  63.7 s Throughput:  16.33 tokens/s Fast forwarding:  1.49 tokens/step
====================================================================================================

Repeating a lot:

USER: tell me a few interesting facts about the sun and the moon.                                                      
ASSISTANT: 1. The sun is a gigantic ball of plasma, while the moonis a rocky body.                                     
2. The sun is approximately 4.6 billion years old, while the moon is approximately 4.5 billion yearsold.                                                                                                                                      
3. The sun is approximately 109 times larger thanthe moon, and the moon is approximately 300 times smaller than thesun.                                                                                                                       
4. The sun is approximately 100 times more massivethan the moon.                                                       
5. The sun is approximately 93 million miles(150 million kilometers) away from Earth, while the moon isapproximately 238,900 miles (384,400 kilometers) away.
6. The sun is approximately 10,000 degrees Fahrenheit (5,538 degreesCelsius) at its surface, while the moon's surface temperature rangesfrom -273 to -233 degrees Fahrenheit (120 to -150 degrees Celsius).
7. Thesun is approximately 864,000 miles (1,392,000 kilometers) in diameter, while the moon isapproximately 2160 miles (3474 kilometers) indiameter.
8. The sun is approximately 109 times more luminous than the moon.                                                     
9. The sun is approximately 10,000 times more massive than the moon.                                                   
10. Thesun is approximately 10,000 times more luminous thanthe moon.         
11. The sun is approximately 10,000 times more massive than the moon.
12. The sun isapproximately 10,000 times more luminous than the moon.
13. The sun is approximately 10,000times more massive than the moon.
14. The sun is approximately 10,000 times more luminous than the moon.
15. The sun is approximately 10,000 times moremassive than the moon.
16. The sun is approximately 10,000 times more luminous than the moon.
17. The sun is approximately 10,000 times more massive thanthe moon.
18. The sun is approximately 10,000 times more luminous than the moon.
19. Thesun is approximately 10,000 times more massive than the moon.
20. The sun is approximately 10,000times more luminous than the moon.
21. The sun isapproximately 10,000 times more massive than the moon.
22. The sun is approximately 10,000 times moreluminous than the moon.
23. The sun is approximately 10,000 times more massive than the moon.
24. The sun is approximately 10,000 times more luminous than the moon.
25. The sun is approximately 10,000 times more massive than the moon.
26. Thesun is approximately 10,000 times more luminous thanthe moon.
27. The sun is approximately 10,000 times more massive than the moon.
28. The sun isapproximately 10,000 times more luminous than the moon.
29. The sun is approximately 10,000times more massive than the moon.
30. The sun is approximately 10,000 times more luminous than the moon.
31. The sun is approximately 10,000 times moremassive than the moon.
32. The sun is approximately 10,000 times more luminous than the moon.
33. The sun is approximately 10,000 times more massive thanthe moon.
34. The sun is approximately 10,000 times more luminous than the moon.
35. Thesun is approximately 10,000 times more massive than the moon.
36. The sun is approximately 10,000times more luminous than the moon.
37. The sun isapproximately 10,000 times more massive than the moon.
38. The sun is approximately 10,000 times moreluminous than the moon.
39. The sun is approximately 10,000 times more massive than the moon.
40. The sun is approximately 10,000 times more luminous than the moon.
41. The sun is approximately 10
======================================SUMMARY=======================================================
Generated tokens:  1040 Time:  47.33 s Throughput:  21.98 tokens/s Fast forwarding:  2.42 tokens/step
====================================================================================================

[snyhlxde1]

Thanks for your interest in our work. Yes, the generation is using greedy decoding. However, there are several potential causes for the discrepancy:

1. error accumulation: even without quantization, it has been observed (for example, from Medusa, https://github.com/FasterDecoding/Medusa/issues/56), where the floating-point error accumulation from CUDA kernels could result in different generation results.

2. quantization: For INT8 quantization, are you using W8A8 (LLM.int8) or other schemes? Due to casting and mix-precision computations, error accumulation could be exacerbated for INT8 quantization. With FP16, generation outputs from our benchmarking are mostly consistent. We might look into integrating PTQ for more efficient inference or PEFT (like QLoRA) for more efficient training as well in the future.

3. Random initialization of n-token sequence: from the Jacobi decoding paper as well as our empirical experiments, the greedy generation outputs should be consistent with AR greedy generation. However, under the hood, since the n-token sequences are randomly initialized every time you run inference, the underlying matrix multiplications have different numerical values and can result in numerical errors.

Re generation speed tradeoff: yes, in general, for open-ended conversational questions, because the generated content is usually more diverse, the set of possible collocations is large, it usually requires more training to obtain more significant speedup. You can try asking coding or math questions (with either the spider checkpoint for text-to-SQL or GSM8K checkpoint for math) where sensible repetitive tokens appear more often to gain more significant speedup.