search

huggingface / tgi-gaudi

Large Language Model Text Generation Inference on Habana Gaudi
http://hf.co/docs/text-generation-inference
Apache License 2.0
27 stars 46 forks source link

ValueError: Unsupported model type t5 #172

Open JunxiChhen opened 4 months ago

JunxiChhen commented 4 months ago

System Info

Linux gaudi2-wsf-test 5.15.0-92-generic #102-Ubuntu SMP Wed Jan 10 09:33:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

image

docker image: ghcr.io/huggingface/tgi-gaudi:2.0.0

Information

Tasks

Reproduction

Doc claims that flan-t5 is supported. However, when I try to run "google/flan-t5-base", shows:

ValueError: Unsupported model type t5

cmd:

docker run --rm  -p 9000:8000 --runtime=habana -e HABANA_VISIBLE_DEVICES=all  -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host ghcr.io/huggingface/tgi-gaudi:2.0.0 --port 8000 --trust-remote-code --model-id google/flan-t5-base --dtype bfloat16 --max-input-tokens 1024 --max-total-tokens 2048

Expected behavior

Either add support of flan-t5 or remove this model from doc.

regisss commented 4 months ago

Doc claims that flan-t5 is supported

Which doc are you talking about? This one: https://huggingface.co/docs/text-generation-inference/supported_models ?

JunxiChhen commented 4 months ago

https://github.com/huggingface/tgi-gaudi/blob/habana-main/docs/source/supported_models.md

regisss commented 4 months ago

Arf yes, this is the source code of TEI's docs which is inherited in this fork. I'll add a banner in the README to specify that this doc doesn't fully apply to this fork.