Models with Phi3Config fail due to missing attention_bias

manikawnth commented 1 month ago

System Info

transformers version: 4.41.1
Platform: Linux-5.15.0-1055-aws-x86_64-with-glibc2.35
Python version: 3.10.14
Huggingface_hub version: 0.23.0
Safetensors version: 0.4.3
Accelerate version: 0.30.1
Accelerate config: not found
PyTorch version (GPU?): 2.3.0 (True)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: yes
Using distributed or parallel set-up in script?: no

Who can help?

No response

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

Running the model using huggingface text-generation-inference server will cause the issue

AttributeError: 'Phi3Config' object has no attribute 'attention_bias'

Reference issue: https://github.com/aws/sagemaker-huggingface-inference-toolkit/issues/123

Expected behavior

tgi server should successfully run the Phi3 model

amyeroberts commented 1 month ago

cc @ArthurZucker @younesbelkada

younesbelkada commented 1 month ago

Hi @manikawnth ! See my comment on the PR, IMO the fix should go on TGI directly, what do you think?

github-actions[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

huggingface / transformers