search

huggingface / text-generation-inference

Large Language Model Text Generation Inference
http://hf.co/docs/text-generation-inference
Apache License 2.0
8.76k stars 1.02k forks source link

AttributeError: 'NoneType' object has no attribute 'replace' #2297

Open almersawi opened 1 month ago

almersawi commented 1 month ago

System Info

Docker image: ghcr.io/huggingface/text-generation-inference:2.2.0-rocm Hardware: AMD MI250

Information

Tasks

Reproduction

  1. run docker run --device /dev/kfd --device /dev/dri ghcr.io/huggingface/text-generation-inference:2.2.0-rocm --model-id TinyLlama/TinyLlama-1.1B-Chat-v1.0
  2. wait until the warmup step

Expected behavior

The model should be deployed as it's officially supported but I get:

File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_causal_lm.py", line 1160, in warmup
    f"tunableop_{MODEL_ID.replace('/', '-')}_tp{self.world_size}_rank{self.rank}.csv",
AttributeError: 'NoneType' object has no attribute 'replace'
ErikKaum commented 1 month ago

Thanks for reporting this and adding the PR 🙌

We're a bit low on bandwidth but can hopefully take a look at it asap 👍

kaustubhrm commented 1 month ago

@almersawi You can continue by disabling PYTORCH_TUNABLEOP feature while starting up. its starting up the server still without warmup. run docker run --device /dev/kfd --device /dev/dri -e PYTORCH_TUNABLEOP_ENABLED=0 ghcr.io/huggingface/text-generation-inference:2.2.0-rocm --model-id TinyLlama/TinyLlama-1.1B-Chat-v1.0