Getting argument device not found error from utils.py

[JulienBeck]

Describe the bug

The chat does not respond.

log shows parameter "device" not found. "/root/miniconda/envs/textgen/lib/python3.11/site-packages/transformers/generation/utils.py "line 1900 -> device=input_ids.device "unexpected keyword argument 'device'"

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

docker container ubuntu:latest (24.04 LTS) with manual installation steps:

FROM ubuntu:latest

RUN apt-get update \ && apt-get install -y build-essential curl unzip python3-pip git \ && mkdir -p /opt/oobabooga/text-generation-webui \ && curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" -o "Miniconda3.sh" \ && bash Miniconda3.sh -b -p $HOME/miniconda \ && export PATH="$HOME/miniconda/bin:$PATH" \ && conda init \ && . /root/.bashrc \ && conda create -n textgen python=3.11 \ && conda activate textgen \ && pip3 install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121 \ && conda install -y -c "nvidia/label/cuda-12.1.1" cuda \ && cd /opt/oobabooga/ \ && git clone https://github.com/oobabooga/text-generation-webui \ && cd text-generation-webui \ && pip install -r requirements.txt

loading any model and starting a chat -> any answer fails with below error

Screenshot

Logs

14:13:44-652554 INFO     Loaded "gpt2" in 7.16 seconds.
14:13:44-653373 INFO     LOADER: "Transformers"
14:13:44-654009 INFO     TRUNCATION LENGTH: 2048
14:13:44-654499 INFO     INSTRUCTION TEMPLATE: "Alpaca"
Traceback (most recent call last):
  File "/opt/oobabooga/text-generation-webui/modules/callbacks.py", line 61, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/oobabooga/text-generation-webui/modules/text_generation.py", line 393, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/root/miniconda/envs/textgen/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda/envs/textgen/lib/python3.11/site-packages/transformers/generation/utils.py", line 1900, in generate
    self._get_logits_warper(generation_config, device=input_ids.device)
TypeError: get_logits_warper_patch() got an unexpected keyword argument 'device'
Output generated in 0.52 seconds (0.00 tokens/s, 0 tokens, context 62, seed 1538071332)

System Info

Ubuntu 24.04
NVIDIA GPU:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.52.04              Driver Version: 555.52.04      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A2                      Off |   00000000:C4:00.0 Off |                    0 |
|  0%   48C    P8              8W /   60W |       1MiB /  15356MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[ZiyaCu]

Same error with llama 3.1 70b awq model

[JulienBeck]

I found the problem, though I do not quite understand why it occured... I created the image usiing

git clone https://github.com/oobabooga/text-generation-webui

which DOES NOT use the current sources. When I do a git pull in /opt/oobabooga/text-generation-webui there are multiple updates. after a pip install -r requirements.txt and a restart of the server, it works as expected.

Hope it helps