win11部署成功，但是LLM回答问题太慢

[xaseven]

win11部署成功，但是LLM回答问题太慢， 问：写一首古诗 控制台回答：{'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '山'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '重'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '水'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '复'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '疑'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '无'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '路'} 但是每行出现的时间间隔很长，cpu和gpu占用率都不高，显卡是4070Ti，12G显存

[qianlei33]

你估计使用cpu，没有使用gpu做推理，安装的时候，需要安装下cuda，在pytorch官网上安装下

[zRzRzRzRzRzRzR]

检查一下pytorch是不是gpu版本，直接按照req装windows是cpu版本，请按照pytorch官网方法装

[panmyuan]

win11部署成功，但是LLM回答问题太慢， 问：写一首古诗 控制台回答：{'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '山'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '重'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '水'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '复'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '疑'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '无'} {'chat_history_id': '554e6dbd602c441e9dac468894772a24', 'text': '路'} 但是每行出现的时间间隔很长，cpu和gpu占用率都不高，显卡是4070Ti，12G显存

可以提供下r你在windows上的requirements.txt文件嘛，官方的各种版本不匹配

[tankGe]

10分钟都没有回答完一首古诗，cpu gpu占用很低 以下是我的安装版本 accelerate 0.30.1 aiofiles 23.2.1 aiohttp 3.9.5 aiosignal 1.3.1 altair 5.3.0 antlr4-python3-runtime 4.9.3 anyio 4.3.0 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arxiv 2.1.0 attrs 23.2.0 auto_gptq 0.7.1 azure-core 1.30.1 azure-storage-blob 12.20.0 backoff 2.2.1 beautifulsoup4 4.12.3 blinker 1.8.2 blis 0.7.11 Brotli 1.1.0 cachetools 5.3.3 catalogue 2.0.10 certifi 2024.2.2 cffi 1.16.0 chardet 5.2.0 charset-normalizer 3.3.2 click 8.1.7 cloudpathlib 0.16.0 colorama 0.4.6 coloredlogs 15.0.1 confection 0.1.4 contourpy 1.2.1 cryptography 42.0.7 cycler 0.12.1 cymem 2.0.8 dashscope 1.19.1 dataclasses-json 0.6.6 datasets 2.19.1 deepdiff 7.0.1 Deprecated 1.2.14 deprecation 2.1.0 dill 0.3.8 distro 1.9.0 duckduckgo-search 3.9.9 effdet 0.4.1 einops 0.7.0 emoji 2.11.1 environs 9.5.0 et-xmlfile 1.1.0 faiss-cpu 1.7.4 fastapi 0.109.0 feedparser 6.0.10 filelock 3.14.0 filetype 1.2.0 flatbuffers 24.3.25 fonttools 4.51.0 frozenlist 1.4.1 fschat 0.2.35 fsspec 2024.3.1 gekko 1.1.1 gitdb 4.0.11 GitPython 3.1.43 google-api-core 2.19.0 google-auth 2.29.0 google-cloud-vision 3.7.2 googleapis-common-protos 1.63.0 greenlet 3.0.3 grpcio 1.63.0 grpcio-status 1.62.2 h11 0.14.0 h2 4.1.0 hpack 4.0.0 httpcore 1.0.5 httpx 0.26.0 httpx-sse 0.4.0 huggingface-hub 0.23.0 humanfriendly 10.0 hyperframe 6.0.1 idna 3.7 importlib_metadata 7.1.0 iniconfig 2.0.0 intel-openmp 2021.4.0 iopath 0.1.10 isodate 0.6.1 Jinja2 3.1.4 joblib 1.4.2 jsonpatch 1.33 jsonpath-python 1.0.6 jsonpointer 2.4 jsonschema 4.22.0 jsonschema-specifications 2023.12.1 kiwisolver 1.4.5 langchain 0.1.6 langchain-community 0.0.38 langchain-core 0.1.52 langchain-experimental 0.0.47 langcodes 3.4.0 langdetect 1.0.9 langsmith 0.1.58 language_data 1.2.0 layoutparser 0.3.4 llama-index 0.9.35 lxml 5.2.2 marisa-trie 1.1.1 Markdown 3.6 markdown-it-py 3.0.0 markdown2 2.4.13 markdownify 0.11.6 MarkupSafe 2.1.5 marshmallow 3.21.2 matplotlib 3.9.0 mdurl 0.1.2 metaphor-python 0.1.23 minio 7.2.7 mkl 2021.4.0 mpmath 1.3.0 msg-parser 1.2.0 multidict 6.0.5 multiprocess 0.70.16 murmurhash 1.0.10 mypy-extensions 1.0.0 nest-asyncio 1.6.0 networkx 3.3 nh3 0.2.17 nltk 3.8.1 numexpr 2.8.6 numpy 1.24.4 olefile 0.47 omegaconf 2.3.0 onnx 1.16.0 onnxruntime 1.15.1 openai 1.30.1 opencv-python 4.9.0.80 openpyxl 3.1.2 optimum 1.19.2 ordered-set 4.1.0 orjson 3.10.3 packaging 23.2 pandas 2.0.3 pathlib 1.0.1 pdf2image 1.17.0 pdfminer.six 20231228 pdfplumber 0.11.0 peft 0.11.0 pikepdf 8.4.1 Pillow 9.5.0 pillow_heif 0.16.0 pip 24.0 pluggy 1.5.0 portalocker 2.8.2 preshed 3.0.9 prompt-toolkit 3.0.43 proto-plus 1.23.0 protobuf 4.25.3 psutil 5.9.8 pyarrow 16.1.0 pyarrow-hotfix 0.6 pyasn1 0.6.0 pyasn1_modules 0.4.0 pyclipper 1.3.0.post5 pycocotools 2.0.7 pycparser 2.22 pycryptodome 3.20.0 pydantic 1.10.13 pydeck 0.9.1 Pygments 2.18.0 PyJWT 2.8.0 pymilvus 2.4.1 PyMuPDF 1.23.16 PyMuPDFb 1.23.9 pypandoc 1.13 pyparsing 3.1.2 pypdf 4.2.0 pypdfium2 4.30.0 pyreadline3 3.4.1 pytesseract 0.3.10 pytest 7.4.3 python-dateutil 2.9.0.post0 python-decouple 3.8 python-docx 1.1.2 python-dotenv 1.0.1 python-iso639 2024.4.27 python-magic 0.4.27 python-magic-bin 0.4.14 python-multipart 0.0.9 python-pptx 0.6.23 pytz 2024.1 pywin32 306 PyYAML 6.0.1 rapidfuzz 3.9.0 rapidocr-onnxruntime 1.3.8 referencing 0.35.1 regex 2024.5.15 requests 2.31.0 rich 13.7.1 rouge 1.0.1 rpds-py 0.18.1 rsa 4.9 safetensors 0.4.3 scikit-learn 1.4.2 scipy 1.13.0 sentence-transformers 2.2.2 sentencepiece 0.2.0 setuptools 69.5.1 setuptools-scm 8.1.0 sgmllib3k 1.0.0 shapely 2.0.4 shortuuid 1.0.13 simplejson 3.19.2 six 1.16.0 smart-open 6.4.0 smmap 5.0.1 sniffio 1.3.1 socksio 1.0.0 soupsieve 2.5 spacy 3.7.2 spacy-legacy 3.0.12 spacy-loggers 1.0.5 SQLAlchemy 2.0.19 srsly 2.4.8 sse-starlette 1.8.2 starlette 0.35.0 streamlit 1.30.0 streamlit-aggrid 0.3.4.post3 streamlit-antd-components 0.3.1 streamlit-chatbox 1.1.11 streamlit-feedback 0.1.3 streamlit_modal 0.1.0 streamlit-option-menu 0.3.12 strsimpy 0.2.1 svgwrite 1.4.3 sympy 1.12 tabulate 0.9.0 tbb 2021.11.0 tenacity 8.3.0 thinc 8.2.3 threadpoolctl 3.5.0 tiktoken 0.5.2 timm 1.0.3 tokenizers 0.15.2 toml 0.10.2 toolz 0.12.1 torch 2.3.0+cu118 torchaudio 2.3.0+cu118 torchvision 0.18.0+cu118 tornado 6.4 tqdm 4.66.1 transformers 4.37.2 transformers-stream-generator 0.0.4 typer 0.9.4 typing_extensions 4.11.0 typing-inspect 0.9.0 tzdata 2024.1 tzlocal 5.2 ujson 5.10.0 unstructured 0.11.0 unstructured-client 0.22.0 unstructured-inference 0.7.15 unstructured.pytesseract 0.3.12 urllib3 2.2.1 uvicorn 0.29.0 validators 0.28.1 wasabi 1.1.2 watchdog 3.0.0 wavedrom 2.0.3.post3 wcwidth 0.2.13 weasel 0.3.4 websockets 12.0 wheel 0.43.0 wrapt 1.16.0 xformers 0.0.23.post1 xlrd 2.0.1 XlsxWriter 3.2.0 xxhash 3.4.1 yarl 1.9.4 youtube-search 2.1.2 zipp 3.18.1