search

xusenlinzy / api-for-open-llm

Openai style api for open large language models, using LLMs just as chatgpt! Support for LLaMA, LLaMA-2, BLOOM, Falcon, Baichuan, Qwen, Xverse, SqlCoder, CodeLLaMA, ChatGLM, ChatGLM2, ChatGLM3 etc. 开源大模型的统一后端接口
Apache License 2.0
2.33k stars 264 forks source link

dify调用chatglm4-chat接口报错500（TypeError: object of type 'int' has no len()） #295

Closed besthong999 closed 3 months ago

besthong999 commented 3 months ago

提交前必须检查以下项目 | The following items must be checked before submission

问题类型 | Type of problem

模型推理和部署 | Model inference and deployment

操作系统 | Operating system

Linux

详细描述问题 | Detailed description of the problem

相关包版本： pip list Package Version

accelerate 0.31.0 aiofiles 23.2.1 aiohttp 3.9.5 aiosignal 1.3.1 altair 5.3.0 annotated-types 0.7.0 antlr4-python3-runtime 4.9.3 anyio 4.4.0 attrs 23.2.0 backoff 2.2.1 beautifulsoup4 4.12.3 bitsandbytes 0.43.1 Brotli 1.0.9 cachetools 5.3.3 certifi 2024.6.2 cffi 1.16.0 chardet 5.2.0 charset-normalizer 3.3.2 click 8.1.7 cloudpickle 3.0.0 cmake 3.29.5.1 coloredlogs 15.0.1 contourpy 1.2.1 cpm-kernels 1.0.11 cryptography 42.0.8 cycler 0.12.1 dataclasses-json 0.6.7 datasets 2.20.0 deepdiff 7.0.1 Deprecated 1.2.14 dill 0.3.8 diskcache 5.6.3 distro 1.9.0 dnspython 2.6.1 effdet 0.4.1 einops 0.8.0 email_validator 2.1.1 emoji 2.12.1 et-xmlfile 1.1.0 fastapi 0.111.0 fastapi-cli 0.0.4 ffmpy 0.3.2 filelock 3.13.1 filetype 1.2.0 flatbuffers 24.3.25 fonttools 4.53.0 frozenlist 1.4.1 fsspec 2024.5.0 gmpy2 2.1.2 google-api-core 2.19.0 google-auth 2.30.0 google-cloud-vision 3.7.2 googleapis-common-protos 1.63.1 gradio 4.36.1 gradio_client 1.0.1 greenlet 3.0.3 grpcio 1.64.1 grpcio-status 1.62.2 h11 0.14.0 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.23.3 humanfriendly 10.0 idna 3.7 importlib_resources 6.4.0 interegular 0.3.3 iopath 0.1.10 Jinja2 3.1.4 joblib 1.4.2 jsonpatch 1.33 jsonpath-python 1.0.6 jsonpointer 3.0.0 jsonschema 4.22.0 jsonschema-specifications 2023.12.1 kiwisolver 1.4.5 langchain 0.2.3 langchain-community 0.2.4 langchain-core 0.2.5 langchain-text-splitters 0.2.1 langdetect 1.0.9 langsmith 0.1.77 lark 1.1.9 layoutparser 0.3.4 llvmlite 0.43.0 lm-format-enforcer 0.10.1 loguru 0.7.2 lxml 5.2.2 Markdown 3.6 markdown-it-py 3.0.0 MarkupSafe 2.1.5 marshmallow 3.21.3 matplotlib 3.9.0 mdurl 0.1.2 mkl-fft 1.3.8 mkl-random 1.2.4 mkl-service 2.4.0 mpmath 1.3.0 msgpack 1.0.8 multidict 6.0.5 multiprocess 0.70.16 mypy-extensions 1.0.0 nest-asyncio 1.6.0 networkx 3.3 ninja 1.11.1.1 nltk 3.8.1 numba 0.60.0 numpy 1.26.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-ml-py 12.555.43 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.5.40 nvidia-nvtx-cu12 12.1.105 olefile 0.47 omegaconf 2.3.0 onnx 1.16.1 onnxruntime 1.18.0 openai 1.34.0 opencv-python 4.10.0.82 openparse 0.5.6 openpyxl 3.1.4 ordered-set 4.1.0 orjson 3.10.4 outlines 0.0.45 packaging 23.2 pandas 2.2.2 pdf2image 1.17.0 pdfminer.six 20231228 pdfplumber 0.11.1 peft 0.11.1 pikepdf 9.0.0 pillow 10.3.0 pillow_heif 0.16.0 pip 24.0 portalocker 2.8.2 prometheus_client 0.20.0 prometheus-fastapi-instrumentator 7.0.0 proto-plus 1.23.0 protobuf 4.25.3 psutil 5.9.8 py-cpuinfo 9.0.0 pyairports 2.1.1 pyarrow 16.1.0 pyarrow-hotfix 0.6 pyasn1 0.6.0 pyasn1_modules 0.4.0 pyclipper 1.3.0.post5 pycocotools 2.0.7 pycountry 24.6.1 pycparser 2.22 pydantic 2.7.4 pydantic_core 2.18.4 pydub 0.25.1 Pygments 2.18.0 PyMuPDF 1.24.5 PyMuPDFb 1.24.3 pypandoc 1.13 pyparsing 3.1.2 pypdf 4.2.0 pypdfium2 4.30.0 PySocks 1.7.1 pytesseract 0.3.10 python-dateutil 2.9.0.post0 python-docx 1.1.2 python-dotenv 1.0.0 python-iso639 2024.4.27 python-magic 0.4.27 python-multipart 0.0.9 python-oxmsg 0.0.1 python-pptx 0.6.23 pytz 2024.1 PyYAML 6.0.1 rapidfuzz 3.9.3 rapidocr-onnxruntime 1.3.22 ray 2.24.0 referencing 0.35.1 regex 2024.5.15 requests 2.32.2 requests-toolbelt 1.0.0 rich 13.7.1 rpds-py 0.18.1 rsa 4.9 ruff 0.4.9 safetensors 0.4.3 scikit-learn 1.5.0 scipy 1.13.1 semantic-version 2.10.0 sentence-transformers 3.0.1 sentencepiece 0.2.0 setuptools 69.5.1 shapely 2.0.4 shellingham 1.5.4 six 1.16.0 sniffio 1.3.1 soupsieve 2.5 SQLAlchemy 2.0.30 sse-starlette 2.1.0 starlette 0.37.2 starlette-context 0.3.6 sympy 1.12 tabulate 0.9.0 tenacity 8.3.0 threadpoolctl 3.5.0 tiktoken 0.7.0 timm 1.0.3 tokenizers 0.19.1 tomlkit 0.12.0 toolz 0.12.1 torch 2.3.0 torchvision 0.18.1 tqdm 4.66.4 transformers 4.40.0 transformers-stream-generator 0.0.5 triton 2.3.0 typer 0.12.3 typing_extensions 4.11.0 typing-inspect 0.9.0 tzdata 2024.1 ujson 5.10.0 unstructured 0.14.5 unstructured-client 0.23.3 unstructured-inference 0.7.33 unstructured.pytesseract 0.3.12 urllib3 2.2.2 uvicorn 0.30.1 uvloop 0.19.0 vllm 0.5.0.post1 vllm-flash-attn 2.5.9 watchfiles 0.22.0 websockets 11.0.3 wheel 0.43.0 wrapt 1.16.0 xformers 0.0.26.post1 xlrd 2.0.1 XlsxWriter 3.2.0 xxhash 3.4.1 yarl 1.9.4

相关配置： `PORT=8002

model related

MODEL_NAME=chatglm4 MODEL_PATH=/data/llm/models/glm-4-9b-chat CONTEXT_LEN= LOAD_IN_8BIT=false LOAD_IN_4BIT=false PROMPT_NAME=chatglm4

rag related

EMBEDDING_NAME= RERANK_NAME=

device related

"auto", "cuda:0", "cuda:1", ...

DEVICE_MAP=cuda:0 GPUS= NUM_GPUs=1 DTYPE=half

api related

API_PREFIX=/v1

USE_STREAMER_V2=false ENGINE=default

TASKS=llm

TASKS=llm,rag`

DEBUG | api.routes.chat:create_chat_completion:56 - ==== request ==== {'model': 'gpt-3.5-turbo', 'frequency_penalty': 0.0, 'function_call': None, 'functions': None, 'logit_bias': None, 'logprobs': False, 'max_tokens': 20, 'n': 1, 'presence_penalty': 0.0, 'response_format': None, 'seed': None, 'stop': ['<|observation|>', '<user>', '<|endoftext|>'], 'temperature': 0.0, 'tool_choice': None, 'tools': None, 'top_logprobs': None, 'top_p': 1.0, 'user': None, 'stream': False, 'repetition_penalty': 1.03, 'typical_p': None, 'watermark': False, 'best_of': 1, 'ignore_eos': False, 'use_beam_search': False, 'stop_token_ids': [151336, 151329, 151338], 'skip_special_tokens': True, 'spaces_between_special_tokens': True, 'min_p': 0.0, 'include_stop_str_in_output': False, 'length_penalty': 1.0, 'guided_json': None, 'guided_regex': None, 'guided_choice': None, 'guided_grammar': None, 'guided_decoding_backend': None, 'prompt_or_messages': [{'role': 'user', 'content': 'ping'}], 'echo': False} INFO: 172.20.0.226:53058 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__ await self.middleware_stack(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 72, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/llm/llama/api-for-open-llm/api/routes/chat.py", line 58, in create_chat_completion iterator_or_completion = await run_in_threadpool(engine.create_chat_completion, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool return await anyio.to_thread.run_sync(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run result = context.run(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/llm/llama/api-for-open-llm/api/engine/hf.py", line 381, in create_chat_completion else self._create_chat_completion(params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/llm/llama/api-for-open-llm/api/engine/hf.py", line 303, in _create_chat_completion for output in self._generate(params): File "/data/llm/llama/api-for-open-llm/api/engine/hf.py", line 102, in _generate inputs = self.template.convert_messages_to_ids( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/llm/llama/api-for-open-llm/api/templates/base.py", line 63, in convert_messages_to_ids input_len = len(token_ids) ^^^^^^^^^^^^^^ TypeError: object of type 'int' has no len()

Dependencies

通过dify版本为0.6.11通过调用glm-4-9b-chat，启动正常但是调用报错。一直提示：TypeError: object of type 'int' has no len() 2024-07-04 16:55:11.956 | DEBUG | api.routes.chat:create_chat_completion:56 - ==== request ==== {'model': 'chatglm3-6b', 'frequency_penalty': 0.0, 'function_call': None, 'functions': None, 'logit_bias': None, 'logprobs': False, 'max_tokens': 16, 'n': 1, 'presence_penalty': 0.0, 'response_format': None, 'seed': None, 'stop': ['<|observation|>', '<|endoftext|>', '<user>'], 'temperature': 0.9, 'tool_choice': None, 'tools': None, 'top_logprobs': None, 'top_p': 1.0, 'user': None, 'stream': True, 'repetition_penalty': 1.03, 'typical_p': None, 'watermark': False, 'best_of': 1, 'ignore_eos': False, 'use_beam_search': False, 'stop_token_ids': [151336, 151329, 151338], 'skip_special_tokens': True, 'spaces_between_special_tokens': True, 'min_p': 0.0, 'include_stop_str_in_output': False, 'length_penalty': 1.0, 'guided_json': None, 'guided_regex': None, 'guided_choice': None, 'guided_grammar': None, 'guided_decoding_backend': None, 'prompt_or_messages': [{'role': 'user', 'content': 'ping'}], 'echo': False} INFO: 172.20.0.226:45114 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__ await self.middleware_stack(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/routing.py", line 72, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/llm/llama/api-for-open-llm/api/routes/chat.py", line 62, in create_chat_completion first_response = await run_in_threadpool(next, iterator_or_completion) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool return await anyio.to_thread.run_sync(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/data/conda/aconda3/envs/glm4/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run result = context.run(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/llm/llama/api-for-open-llm/api/engine/hf.py", line 209, in _create_chat_completion_stream for i, output in enumerate(self._generate(params)): File "/data/llm/llama/api-for-open-llm/api/engine/hf.py", line 102, in _generate inputs = self.template.convert_messages_to_ids( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/llm/llama/api-for-open-llm/api/templates/base.py", line 63, in convert_messages_to_ids input_len = len(token_ids) ^^^^^^^^^^^^^^ TypeError: object of type 'int' has no len()

运行日志或截图 | Runtime logs or screenshots

TypeError: object of type 'int' has no len() 1A581960-C168-4425-8AAC-AB72A15CD994

xusenlinzy commented 3 months ago

刚注意到glm4的tokenizer文件改动了，更新一下项目代码应该就可以了

besthong999 commented 3 months ago

你好已经是最新才报这个错误。

---- 回复的原邮件 ---- | 发件人 | @.> | | 日期 | 2024年07月05日 10:16 | | 收件人 | @.> | | 抄送至 | @.>@.> | | 主题 | Re: [xusenlinzy/api-for-open-llm] dify调用chatglm4-chat接口报错500（TypeError: object of type 'int' has no len()） (Issue #295) |

刚注意到glm4的tokenizer文件改动了，更新一下项目代码应该就可以了

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

xusenlinzy commented 3 months ago

是这个版本吗https://github.com/xusenlinzy/api-for-open-llm/commit/8dd1db9ee284376fad4c10e60951e6f114af4d40

besthong999 commented 2 months ago

升级后可以了，接入dify没有调用工具类，这块是需要dify支持吗？

---- 回复的原邮件 ---- | 发件人 | @.> | | 日期 | 2024年07月05日 11:43 | | 收件人 | @.> | | 抄送至 | @.>@.> | | 主题 | Re: [xusenlinzy/api-for-open-llm] dify调用chatglm4-chat接口报错500（TypeError: object of type 'int' has no len()） (Issue #295) |

是这个版本吗8dd1db9

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

besthong999 commented 2 months ago

你好，我再次试了下，目前启动正常，但是chatGLM4的工具类没有生效。dify已升级到最新版本为：Version 0.6.12-fix1，chatGLM4也是最新版本： 但是在调用dify的过程中发现工具类未生效。

启动信息如下： { "model_name": "chatglm4", "model_path": "/data/llm/llama/models/glm-4-9b-chat", "dtype": "bfloat16", "load_in_8bit": false, "load_in_4bit": false, "context_length": -1, "chat_template": "chatglm4", "rope_scaling": null, "flash_attn": false, "interrupt_requests": true, "host": "0.0.0.0", "port": 8000, "api_prefix": "/v1", "engine": "default", "tasks": [ "llm" ], "device_map": "cuda:0", "gpus": null, "num_gpus": 1, "activate_inference": true, "model_names": [ "chatglm4" ], "api_keys": null }

请求记录 {'model': 'gpt-4', 'frequency_penalty': 0.5, 'function_call': None, 'functions': [{'name': 'aitoolsqueryServiceProviderInfo_post', 'description': '服务商联系方式查询，可以查询服务商信息、目的国家电话、服务商电话、邮局电话、单号相关联系电话、派送方的联系方式、末端电话', 'parameters': {'type': 'object', 'properties': {'orderNo': {'type': 'string', 'description': '单号'}}, 'required': ['orderNo']}}, {'name': 'aitoolsqueryChangeNumberException_get', 'description': '换号信息查询', 'parameters': {'type': 'object', 'properties': {'orderNo': {'type': 'string', 'description': '单号'}}, 'required': ['orderNo']}}, {'name': 'aitoolsqueryAllException_get', 'description': '订单问题查询，可以查询 订单的问题件、库内异常', 'parameters': {'type': 'object', 'properties': {'orderNo': {'type': 'string', 'description': '单号'}}, 'required': []}}, {'name': 'getMailitemByOrderNo', 'description': '订单查询', 'parameters': {'type': 'object', 'properties': {'orderNo': {'type': 'string', 'description': '4px单号'}}, 'required': ['orderNo']}}, {'name': 'dataset_6f89cbad_382c_40e9_b5ee_df1f15ec6ed8', 'description': '走货咨询&系统问题知识', 'parameters': {'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'Query for the dataset to be used to retrieve the dataset.'}}, 'required': ['query']}}], 'logit_bias': None, 'logprobs': False, 'max_tokens': 1024, 'n': 1, 'presence_penalty': 0.5, 'response_format': None, 'seed': None, 'stop': ['', '<|observation|>', '<|endoftext|>'], 'temperature': 0.2, 'tool_choice': None, 'tools': None, 'top_logprobs': None, 'top_p': 0.75, 'user': '7a8d2ffb-42b9-432b-bc88-0fa108c389cf', 'stream': True, 'repetition_penalty': 1.03, 'typical_p': None, 'watermark': False, 'best_of': 1, 'ignore_eos': False, 'use_beam_search': False, 'stop_token_ids': [151336, 151329, 151338], 'skip_special_tokens': True, 'spaces_between_special_tokens': True, 'min_p': 0.0, 'include_stop_str_in_output': False, 'length_penalty': 1.0, 'guided_json': None, 'guided_regex': None, 'guided_choice': None, 'guided_grammar': None, 'guided_decoding_backend': None, 'prompt_or_messages': [{'role': 'system', 'content': '# 角色\n你是一个客服智能助手专家，我是客服，请记住，你的名字是小智。\n## 技能\n- 回答问题：根据用户的问题，提供准确、清晰、有用的答案。\n\n## 限制\n- 如果文档与问题无关，根据你的知识回答。\n- 所输出的内容必须按照给定的格式进行组织，不能偏离框架要求。\n- 如果问题中不存在orderNo参数则禁止使用工具。\n- 不要组装orderNo参数。\n'}, {'role': 'user', 'content': '4PX3000001314912CN\n\n'}, {'role': 'assistant', 'content': '\n您好，根据您提供的订单号4PX3000001314912CN，我将为您查询订单详情。请稍等片刻。'}, {'role': 'user', 'content': '结果呢？'}, {'role': 'assistant', 'content': '\n很抱歉，但作为一个AI，我无法直接访问外部数据库或系统来检索实时订单信息。因此，我无法提供关于订单号4PX300000131491的具体结果。\n\n为了获取订单状态和详情，建议您直接登录到4PX的官方网站或联系4PX的客户服务部门，他们能够根据您的订单号提供最新的信息。如果您需要帮助了解如何操作或联系他们，请告诉我，我会尽力提供指导。'}, {'role': 'user', 'content': '4PX3000001314912\n'}], 'echo': False} INFO: 10.10.107.50:51800 - "POST /v1/chat/completions HTTP/1.1" 200 OK 2024-07-08 23:52:54.644 | DEBUG | api.routes.chat:create_chat_completion:56 - ==== request ==== {'model': 'gpt-3.5-turbo', 'frequency_penalty': 0.0, 'function_call': None, 'functions': None, 'logit_bias': None, 'logprobs': False, 'max_tokens': 256, 'n': 1, 'presence_penalty': 0.0, 'response_format': None, 'seed': None, 'stop': ['', '<|observation|>', '<|endoftext|>'], 'temperature': 0.0, 'tool_choice': None, 'tools': None, 'top_logprobs': None, 'top_p': 1.0, 'user': None, 'stream': False, 'repetition_penalty': 1.03, 'typical_p': None, 'watermark': False, 'best_of': 1, 'ignore_eos': False, 'use_beam_search': False, 'stop_token_ids': [151336, 151329, 151338], 'skip_special_tokens': True, 'spaces_between_special_tokens': True, 'min_p': 0.0, 'include_stop_str_in_output': False, 'length_penalty': 1.0, 'guided_json': None, 'guided_regex': None, 'guided_choice': None, 'guided_grammar': None, 'guided_decoding_backend': None, 'prompt_or_messages': [{'role': 'user', 'content': 'Human: 4PX3000001314\n\n\nAssistant: 您好，根据您提供的订单号4PX3000001，我将为您查询订单详情。请稍等片刻。\nHuman: 结果呢？\nAssistant: 很抱歉，但作为一个AI，我无法直接访问外部数据库或系统来检索实时订单信息。因此，我无法提供关于订单号4PX3000001314的具体结果。\n\n为了获取订单状态和详情，建议您直接登录到4PX的官方网站或联系4PX的客户服务部门，他们能够根据您的订单号提供最新的信息。如果您需要帮助了解如何操作或联系他们，请告诉我，我会尽力提供指导。\nHuman: 4PX3000001314\n\nAssistant: 很抱歉，但作为一个AI，我无法直接访问外部数据库或系统来检索实时订单信息。因此，我无法验证订单号4PX30000013的具体状态或详情。\n\n如果您需要查询这个订单的信息，以下是一些可能的步骤：\n\n1. 登录4PX账户：如果您有4PX的账户，请尝试登录并查看订单历史。\n2. 联系4PX客服：通过4PX的客服热线或在线客服系统联系他们，提供订单号进行查询。\n3. 检查电子邮件：有时候订单更新会通过电子邮件发送，请检查您的邮箱。\n\n如果您需要进一步的帮助，请提供更多的上下文或者说明您需要什么样的帮助。\nPlease help me predict the three most likely questions that human would ask, and keeping each question under 20 characters.\nThe output must be an array in JSON format following the specified schema:\n["question1","question2","question3"]\n\nquestions:\n'}], 'echo': False}

以上问题，麻烦看下~

陈树宏

@. | ---- 回复的原邮件 ---- | 发件人 | @.> | | 发送日期 | 2024年7月8日 09:44 | | 收件人 | @.> | | 抄送人 | @.>, @.***> | | 主题 | 回复：[xusenlinzy/api-for-open-llm] dify调用chatglm4-chat接口报错500（TypeError: object of type 'int' has no len()） (Issue #295) | 升级后可以了，接入dify没有调用工具类，这块是需要dify支持吗？

---- 回复的原邮件 ---- | 发件人 | @.> | | 日期 | 2024年07月05日 11:43 | | 收件人 | @.> | | 抄送至 | @.>@.> | | 主题 | Re: [xusenlinzy/api-for-open-llm] dify调用chatglm4-chat接口报错500（TypeError: object of type 'int' has no len()） (Issue #295) |

是这个版本吗8dd1db9

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>