fix: quantization with DeepSpeed HE

microsoft / DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

https://www.deepspeed.ai/

Apache License 2.0

33.63k stars 3.95k forks source link

fix: quantization with DeepSpeed HE #5624

Open Atry opened 3 weeks ago

Atry commented 3 weeks ago

When the model is quantized, the hidden sizes cannot be determined from ds_shape and shape, because they are 1 dimensional. This PR fixes the bug by determining hidden sizes from in_features and out_features.

This PR fixes #5398

Atry commented 2 weeks ago

@lekurile The CI error is transient:

huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/models?cursor=eyJfaWQiOnsiJGd0IjoiNjYzNGRlNzA2MzRhNjdjNGVmMmE4YWEwIn19

Could you retry CI?

loadams commented 4 days ago

@lekurile The CI error is transient:

huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/models?cursor=eyJfaWQiOnsiJGd0IjoiNjYzNGRlNzA2MzRhNjdjNGVmMmE4YWEwIn19

Could you retry CI?

Retrying CI.

Atry commented 1 day ago

@lekurile The CI error is transient:

huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/models?cursor=eyJfaWQiOnsiJGd0IjoiNjYzNGRlNzA2MzRhNjdjNGVmMmE4YWEwIn19

Could you retry CI?

Retrying CI.

The error is still HTTP 429