Closed Qubitium closed 5 months ago
hmmm, please start the test with our deepspeed-mii: https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-fp6/03-05-2024/README.md#4-how-to-begin-with-deepspeed-fp6--
@Qubitium - can you please share your pydantic version from pip list
?
Describe the bug
Unable to use/test fp6 quantization in deepspeed 0.14 in inference mode on a GPT2 model. There is little documentation on usage right so not sure if I have the wrong init method. I am passing quant based on the changes on the commited fp6 PR. by @loadams
To Reproduce
Crash Error
Code
But DeepSpeedInferenceConfig code has
quant
field:Expected behavior
Load success
ds_report output
System info (please complete the following information):