search

Atinoda / text-generation-webui-docker

Docker variants of oobabooga's text-generation-webui, including pre-built images.
GNU Affero General Public License v3.0
393 stars 78 forks source link

Default Nvidia TensorRT-LLM variant v1.14 does not start #59

Open ht-sp opened 2 weeks ago

ht-sp commented 2 weeks ago

Greetings, thank you for your work. After successful running the nvidia default variant, I tried to run the TensorRT variant (default-nvidia-tensorrtllm). But it does not start. No modifications to the image/s or compose file (except for changing the variant). The older image versions (<1.14) do work as expected.

System: memory: 64GiB System Memory processor: 13th Gen Intel(R) Core(TM) i5-13500 display: AD104GL [RTX 4000 SFF Ada Generation]

text-generation-webui  | === Running text-generation-webui variant: 'Nvidia Extended (TensorRT-LLM)' v1.14 ===
text-generation-webui  | === (This version is 4 commits behind origin main) ===
text-generation-webui  | === Image build date: 2024-08-20 21:23:03 ===
text-generation-webui  | ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
text-generation-webui  | │ /app/server.py:40 in <module>                                                │
text-generation-webui  | │                                                                              │
text-generation-webui  | │    39 import modules.extensions as extensions_module                         │
text-generation-webui  | │ ❱  40 from modules import (                                                  │
text-generation-webui  | │    41     chat,                                                              │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /app/modules/training.py:21 in <module>                                      │
text-generation-webui  | │                                                                              │
text-generation-webui  | │    20 from datasets import Dataset, load_dataset                             │
text-generation-webui  | │ ❱  21 from peft import (                                                     │
text-generation-webui  | │    22     LoraConfig,                                                        │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /venv/lib/python3.10/site-packages/peft/__init__.py:22 in <module>           │
text-generation-webui  | │                                                                              │
text-generation-webui  | │    21                                                                        │
text-generation-webui  | │ ❱  22 from .auto import (                                                    │
text-generation-webui  | │    23     AutoPeftModel,                                                     │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /venv/lib/python3.10/site-packages/peft/auto.py:32 in <module>               │
text-generation-webui  | │                                                                              │
text-generation-webui  | │    31 from .config import PeftConfig                                         │
text-generation-webui  | │ ❱  32 from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING                  │
text-generation-webui  | │    33 from .peft_model import (                                              │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /venv/lib/python3.10/site-packages/peft/mapping.py:22 in <module>            │
text-generation-webui  | │                                                                              │
text-generation-webui  | │    21                                                                        │
text-generation-webui  | │ ❱  22 from peft.tuners.xlora.model import XLoraModel                         │
text-generation-webui  | │    23                                                                        │
text-generation-webui  | │                                                                              │
text-generation-webui  | │                           ... 2 frames hidden ...                            │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /venv/lib/python3.10/site-packages/peft/tuners/lora/model.py:50 in <module>  │
text-generation-webui  | │                                                                              │
text-generation-webui  | │    49 from .aqlm import dispatch_aqlm                                        │
text-generation-webui  | │ ❱  50 from .awq import dispatch_awq                                          │
text-generation-webui  | │    51 from .config import LoraConfig                                         │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /venv/lib/python3.10/site-packages/peft/tuners/lora/awq.py:26 in <module>    │
text-generation-webui  | │                                                                              │
text-generation-webui  | │    25 if is_auto_awq_available():                                            │
text-generation-webui  | │ ❱  26     from awq.modules.linear import WQLinear_GEMM                       │
text-generation-webui  | │    27                                                                        │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /venv/lib/python3.10/site-packages/awq/__init__.py:2 in <module>             │
text-generation-webui  | │                                                                              │
text-generation-webui  | │   1 __version__ = "0.2.6"                                                    │
text-generation-webui  | │ ❱ 2 from awq.models.auto import AutoAWQForCausalLM                           │
text-generation-webui  | │   3                                                                          │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /venv/lib/python3.10/site-packages/awq/models/__init__.py:21 in <module>     │
text-generation-webui  | │                                                                              │
text-generation-webui  | │   20 from .llava_next import LlavaNextAWQForCausalLM                         │
text-generation-webui  | │ ❱ 21 from .phi3 import Phi3AWQForCausalLM                                    │
text-generation-webui  | │   22 from .cohere import CohereAWQForCausalLM                                │
text-generation-webui  | │                                                                              │
text-generation-webui  | │ /venv/lib/python3.10/site-packages/awq/models/phi3.py:7 in <module>          │
text-generation-webui  | │                                                                              │
text-generation-webui  | │     6 from awq.modules.fused.model import Phi3Model as AWQPhi3Model          │
text-generation-webui  | │ ❱   7 from transformers.models.phi3.modeling_phi3 import (                   │
text-generation-webui  | │     8     Phi3DecoderLayer as OldPhi3DecoderLayer,                           │
text-generation-webui  | ╰──────────────────────────────────────────────────────────────────────────────╯
text-generation-webui  | ModuleNotFoundError: No module named 'transformers.models.phi3'
text-generation-webui exited with code 1
Atinoda commented 1 week ago

Hi @ht-sp, thank you for letting me know. I have updated some dependencies in the latest image - can you please check and see if it's working now?