Aries-ckt
commented
9 months ago Search before asking
- [x] I had searched in the issues and found no similar issues.
Operating system information
Linux
Python version information
=3.11
DB-GPT version
main
Related scenes
- [ ] Chat Data
- [ ] Chat Excel
- [ ] Chat DB
- [x] Chat Knowledge
- [ ] Model Management
- [ ] Dashboard
- [ ] Plugins
Installation Information
- [x] Installation From Source
- [ ] Docker Installation
- [ ] Docker Compose Installation
- [ ] Cluster Installation
- [ ] AutoDL Image
- [ ] Other
Device information
GPU: 1
Models information
proxyllm
What happened
单独使用大模型对话可正常使用，上传文档加载进知识库后(milvus向量库+mysql元数据库都有记录，文件是docx后缀格式，文本文件来源于腾讯云官方技术文档下载)，基于知识库问答报错，如下。 ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/applications.py", line 290, in call await super().call(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/applications.py", line 122, in call await self.middleware_stack(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in call raise exc File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in call await self.app(scope, receive, _send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in call await self.simple_response(scope, receive, send, request_headers=headers) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in call raise exc File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in call await self.app(scope, receive, sender) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in call raise e File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in call await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 718, in call await route.handle(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 69, in app await response(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 270, in call async with anyio.create_task_group() as task_group: File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 597, in aexit raise exceptions[0] File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 273, in wrap await func() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 262, in stream_response async for chunk in self.body_iterator: File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/openapi/api_v1/api_v1.py", line 445, in stream_generator async for chunk in chat.stream_call(): File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 62, in stream_call input_values = self.generate_input_values() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 92, in generate_input_values set([os.path.basename(d.metadata.get("source")) for d in docs]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 92, in set([os.path.basename(d.metadata.get("source")) for d in docs]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 142, in basename TypeError: expected str, bytes or os.PathLike object, not NoneType
What you expected to happen
1、为什么docx文档的metadata没有被存入； 2、metadata为空时，是否可以进行异常处置？
How to reproduce
配置.env后，起docker服务(mysql+milvus)后，初始化mysql库表后，python pilot/server/dbgpt_server.py后，先报错如下（1部分），运行原生llm对话没有问题，加载本地.docx文档后，基于知识库问答报错如下（2部分）。
- 1 部分
INFO: 127.0.0.1:56910 - "POST /api/controller/heartbeat HTTP/1.1" 200 OK 2023-09-26 22:13:13 | INFO | db_summary | Vector store name EXAMPLE_1_profile exist 2023-09-26 22:13:13 | INFO | db_summary | init db profile success... 2023-09-26 22:13:13 | INFO | db_summary | db summary embedding success 2023-09-26 22:13:13 | INFO | db_summary | Vector store name history_profile exist 2023-09-26 22:13:13 | INFO | db_summary | init db profile success... 2023-09-26 22:13:13 | INFO | db_summary | db summary embedding success 2023-09-26 22:13:13 | INFO | db_summary | Vector store name prompt_management_profile exist 2023-09-26 22:13:13 | INFO | db_summary | init db profile success... 2023-09-26 22:13:13 | INFO | db_summary | db summary embedding success --- Logging error --- Traceback (most recent call last): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in __init__ self._dbapi_connection = engine.raw_connection() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection return self.pool.connect() ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452, in connect return _ConnectionFairy._checkout(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1267, in _checkout fairy = _ConnectionRecord.checkout(pool) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 716, in checkout rec = pool._do_get() ^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get with util.safe_reraise(): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__ raise exc_value.with_traceback(exc_tb) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get return self._create_connection() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection return _ConnectionRecord(self) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 678, in __init__ self.__connect() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 902, in __connect with util.safe_reraise(): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__ raise exc_value.with_traceback(exc_tb) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 898, in __connect self.dbapi_connection = connection = pool._invoke_creator(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 637, in connect return dialect.connect(*cargs, **cparams) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 615, in connect return self.loaded_dbapi.connect(*cargs, **cparams) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 358, in __init__ self.connect() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 664, in connect self._request_authentication() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 968, in _request_authentication auth_packet = self._process_auth(plugin_name, auth_packet) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 1003, in _process_auth return _auth.sha256_password_auth(self, auth_packet) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/_auth.py", line 187, in sha256_password_auth return _roundtrip(conn, data) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/_auth.py", line 120, in _roundtrip pkt = conn._read_packet() ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 772, in _read_packet packet.raise_for_error() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.OperationalError: (1045, "Access denied for user 'wwt'@'172.17.0.1' (using password: NO)") The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/summary/db_summary_client.py", line 164, in init_db_summary self.db_summary_embedding(item["db_name"], item["db_type"]) File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/summary/db_summary_client.py", line 38, in db_summary_embedding db_summary_client = RdbmsSummary(dbname, db_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/summary/rdbms_db_summary.py", line 21, in __init__ self.db = CFG.LOCAL_DB_MANAGE.get_connect(name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/connections/manages/connection_manager.py", line 145, in get_connect return connect_instance.from_uri_db( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/connections/rdbms/base.py", line 108, in from_uri_db return cls.from_uri(db_url, engine_args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/connections/rdbms/base.py", line 116, in from_uri return cls(create_engine(database_uri, **_engine_args), **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/connections/rdbms/base.py", line 61, in __init__ self._inspector = inspect(engine) ^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/inspection.py", line 145, in inspect ret = reg(subject) ^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 303, in _engine_insp return Inspector._construct(Inspector._init_engine, bind) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 236, in _construct init(self, bind) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 247, in _init_engine engine.connect().close() ^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3264, in connect return self._connection_cls(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 147, in __init__ Connection._handle_dbapi_exception_noconnection( File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2426, in _handle_dbapi_exception_noconnection raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in __init__ self._dbapi_connection = engine.raw_connection() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection return self.pool.connect() ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452, in connect return _ConnectionFairy._checkout(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1267, in _checkout fairy = _ConnectionRecord.checkout(pool) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 716, in checkout rec = pool._do_get() ^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get with util.safe_reraise(): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__ raise exc_value.with_traceback(exc_tb) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get return self._create_connection() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection return _ConnectionRecord(self) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 678, in __init__ self.__connect() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 902, in __connect with util.safe_reraise(): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__ raise exc_value.with_traceback(exc_tb) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 898, in __connect self.dbapi_connection = connection = pool._invoke_creator(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 637, in connect return dialect.connect(*cargs, **cparams) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 615, in connect return self.loaded_dbapi.connect(*cargs, **cparams) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 358, in __init__ self.connect() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 664, in connect self._request_authentication() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 968, in _request_authentication auth_packet = self._process_auth(plugin_name, auth_packet) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 1003, in _process_auth return _auth.sha256_password_auth(self, auth_packet) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/_auth.py", line 187, in sha256_password_auth return _roundtrip(conn, data) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/_auth.py", line 120, in _roundtrip pkt = conn._read_packet() ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 772, in _read_packet packet.raise_for_error() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'wwt'@'172.17.0.1' (using password: NO)") (Background on this error at: https://sqlalche.me/e/20/e3q8) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/logging/__init__.py", line 1110, in emit msg = self.format(record) ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/logging/__init__.py", line 953, in format return fmt.format(record) ^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/logging/__init__.py", line 687, in format record.message = record.getMessage() ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/logging/__init__.py", line 377, in getMessage msg = msg % self.args ~~~~^~~~~~~~~~~ TypeError: not all arguments converted during string formatting Call stack: File "/home/wwt/anaconda3/envs/tops/lib/python3.11/threading.py", line 995, in _bootstrap self._bootstrap_inner() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/summary/db_summary_client.py", line 166, in init_db_summary logger.warn( Message: 'mysql_default_sqlite, mysql summary error!(pymysql.err.OperationalError) (1045, "Access denied for user \'wwt\'@\'172.17.0.1\' (using password: NO)")\n(Background on this error at: https://sqlalche.me/e/20/e3q8)' Arguments: (OperationalError('(pymysql.err.OperationalError) (1045, "Access denied for user \'wwt\'@\'172.17.0.1\' (using password: NO)")'),)
- 2部分
ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/applications.py", line 290, in __call__ await super().__call__(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__ await self.middleware_stack(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__ raise exc File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__ await self.app(scope, receive, _send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in __call__ await self.simple_response(scope, receive, send, request_headers=headers) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__ raise exc File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__ await self.app(scope, receive, sender) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__ raise e File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__ await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__ await route.handle(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 69, in app await response(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 270, in __call__ async with anyio.create_task_group() as task_group: File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__ raise exceptions[0] File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 273, in wrap await func() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 262, in stream_response async for chunk in self.body_iterator: File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/openapi/api_v1/api_v1.py", line 445, in stream_generator async for chunk in chat.stream_call(): File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 62, in stream_call input_values = self.generate_input_values() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 92, in generate_input_values set([os.path.basename(d.metadata.get("source")) for d in docs]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 92, in <listcomp> set([os.path.basename(d.metadata.get("source")) for d in docs]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen posixpath>", line 142, in basename TypeError: expected str, bytes or os.PathLike object, not NoneTypeAdditional context
.docx文档来源可从此处获取：https://cloud.tencent.com/privatecloud/
Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
界面点进Details看看metadata有没有source字段看看呢
IamWWT
github-actions[bot]
Search before asking
Operating system information
Linux
Python version information
DB-GPT version
main
Related scenes
Installation Information
[X] Installation From Source
[ ] Docker Installation
[ ] Docker Compose Installation
[ ] Cluster Installation
[ ] AutoDL Image
[ ] Other
Device information
GPU: 1
Models information
proxyllm
What happened
单独使用大模型对话可正常使用，上传文档加载进知识库后(milvus向量库+mysql元数据库都有记录，文件是docx后缀格式，文本文件来源于腾讯云官方技术文档下载)，基于知识库问答报错，如下。 ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/applications.py", line 290, in call await super().call(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/applications.py", line 122, in call await self.middleware_stack(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in call raise exc File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in call await self.app(scope, receive, _send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in call await self.simple_response(scope, receive, send, request_headers=headers) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in call raise exc File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in call await self.app(scope, receive, sender) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in call raise e File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in call await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 718, in call await route.handle(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/routing.py", line 69, in app await response(scope, receive, send) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 270, in call async with anyio.create_task_group() as task_group: File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 597, in aexit raise exceptions[0] File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 273, in wrap await func() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/starlette/responses.py", line 262, in stream_response async for chunk in self.body_iterator: File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/openapi/api_v1/api_v1.py", line 445, in stream_generator async for chunk in chat.stream_call(): File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 62, in stream_call input_values = self.generate_input_values() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 92, in generate_input_values set([os.path.basename(d.metadata.get("source")) for d in docs]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/scene/chat_knowledge/v1/chat.py", line 92, in
set([os.path.basename(d.metadata.get("source")) for d in docs])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 142, in basename
TypeError: expected str, bytes or os.PathLike object, not NoneType
What you expected to happen
1、为什么docx文档的metadata没有被存入； 2、metadata为空时，是否可以进行异常处置？
How to reproduce
配置.env后，起docker服务(mysql+milvus)后，初始化mysql库表后，python pilot/server/dbgpt_server.py后，先报错如下（1部分），运行原生llm对话没有问题，加载本地.docx文档后，基于知识库问答报错如下（2部分）。
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/summary/db_summary_client.py", line 164, in init_db_summary self.db_summary_embedding(item["db_name"], item["db_type"]) File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/summary/db_summary_client.py", line 38, in db_summary_embedding db_summary_client = RdbmsSummary(dbname, db_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/summary/rdbms_db_summary.py", line 21, in init self.db = CFG.LOCAL_DB_MANAGE.get_connect(name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/connections/manages/connection_manager.py", line 145, in get_connect return connect_instance.from_uri_db( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/connections/rdbms/base.py", line 108, in from_uri_db return cls.from_uri(db_url, engine_args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/connections/rdbms/base.py", line 116, in from_uri return cls(create_engine(database_uri, _engine_args), kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/Downloads/aigc/proj/ops/dtops/pilot/connections/rdbms/base.py", line 61, in init self._inspector = inspect(engine) ^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/inspection.py", line 145, in inspect ret = reg(subject) ^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 303, in _engine_insp return Inspector._construct(Inspector._init_engine, bind) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 236, in _construct init(self, bind) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 247, in _init_engine engine.connect().close() ^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3264, in connect return self._connection_cls(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 147, in init Connection._handle_dbapi_exception_noconnection( File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2426, in _handle_dbapi_exception_noconnection raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in init self._dbapi_connection = engine.raw_connection() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection return self.pool.connect() ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452, in connect return _ConnectionFairy._checkout(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1267, in _checkout fairy = _ConnectionRecord.checkout(pool) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 716, in checkout rec = pool._do_get() ^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get with util.safe_reraise(): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in exit raise exc_value.with_traceback(exc_tb) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get return self._create_connection() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection return _ConnectionRecord(self) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 678, in init self.connect() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 902, in connect with util.safe_reraise(): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in exit raise exc_value.with_traceback(exc_tb) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 898, in __connect self.dbapi_connection = connection = pool._invoke_creator(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 637, in connect return dialect.connect(*cargs, *cparams) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 615, in connect return self.loaded_dbapi.connect(cargs, cparams) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 358, in init self.connect() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 664, in connect self._request_authentication() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 968, in _request_authentication auth_packet = self._process_auth(plugin_name, auth_packet) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 1003, in _process_auth return _auth.sha256_password_auth(self, auth_packet) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/_auth.py", line 187, in sha256_password_auth return _roundtrip(conn, data) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/_auth.py", line 120, in _roundtrip pkt = conn._read_packet() ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/connections.py", line 772, in _read_packet packet.raise_for_error() File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/home/wwt/anaconda3/envs/tops/lib/python3.11/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'wwt'@'172.17.0.1' (using password: NO)") (Background on this error at: https://sqlalche.me/e/20/e3q8)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/wwt/anaconda3/envs/tops/lib/python3.11/logging/init.py", line 1110, in emit msg = self.format(record) ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/logging/init.py", line 953, in format return fmt.format(record) ^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/logging/init.py", line 687, in format record.message = record.getMessage() ^^^^^^^^^^^^^^^^^^^ File "/home/wwt/anaconda3/envs/tops/lib/python3.11/logging/init.py", line 377, in getMessage msg = msg % self.args