search

opea-project / GenAIComps

GenAI components at micro-service level; GenAI service composer to create mega-service
Apache License 2.0
74 stars 136 forks source link

opea/llm-docsum-tgi fails when parse false streaming setting #715

Closed hteeyeoh closed 1 month ago

hteeyeoh commented 1 month ago

When run the comps/llms/summarization/tgi/langchain docker container, when pass the "streaming": false parameter in the curl request:

curl http://${your_ip}:9000/v1/chat/docsum -X POST -d '{"query":"Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5.", "streaming":false}' -H 'Content-Type: application/json'

it returns "Internal Server Error" and the following logs:

llm-docsum-tgi-server  | INFO:     10.223.24.231:41602 - "POST /v1/chat/docsum HTTP/1.1" 500 Internal Server Error
llm-docsum-tgi-server  | ERROR:    Exception in ASGI application
llm-docsum-tgi-server  | Traceback (most recent call last):
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
llm-docsum-tgi-server  |     result = await app(  # type: ignore[func-returns-value]
llm-docsum-tgi-server  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
llm-docsum-tgi-server  |     return await self.app(scope, receive, send)
llm-docsum-tgi-server  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
llm-docsum-tgi-server  |     await super().__call__(scope, receive, send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
llm-docsum-tgi-server  |     await self.middleware_stack(scope, receive, send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
llm-docsum-tgi-server  |     raise exc
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
llm-docsum-tgi-server  |     await self.app(scope, receive, _send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 174, in __call__
llm-docsum-tgi-server  |     raise exc
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 172, in __call__
llm-docsum-tgi-server  |     await self.app(scope, receive, send_wrapper)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
llm-docsum-tgi-server  |     await self.app(scope, receive, send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
llm-docsum-tgi-server  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
llm-docsum-tgi-server  |     raise exc
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
llm-docsum-tgi-server  |     await app(scope, receive, sender)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
llm-docsum-tgi-server  |     await self.middleware_stack(scope, receive, send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
llm-docsum-tgi-server  |     await route.handle(scope, receive, send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
llm-docsum-tgi-server  |     await self.app(scope, receive, send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
llm-docsum-tgi-server  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
llm-docsum-tgi-server  |     raise exc
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
llm-docsum-tgi-server  |     await app(scope, receive, sender)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
llm-docsum-tgi-server  |     response = await f(request)
llm-docsum-tgi-server  |                ^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
llm-docsum-tgi-server  |     raw_response = await run_endpoint_function(
llm-docsum-tgi-server  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
llm-docsum-tgi-server  |     return await run_in_threadpool(dependant.call, **values)
llm-docsum-tgi-server  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
llm-docsum-tgi-server  |     return await anyio.to_thread.run_sync(func, *args)
llm-docsum-tgi-server  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
llm-docsum-tgi-server  |     return await get_async_backend().run_sync_in_worker_thread(
llm-docsum-tgi-server  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2357, in run_sync_in_worker_thread
llm-docsum-tgi-server  |     return await future
llm-docsum-tgi-server  |            ^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 864, in run
llm-docsum-tgi-server  |     result = context.run(func, *args)
llm-docsum-tgi-server  |              ^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/comps/llms/summarization/tgi/langchain/llm.py", line 72, in llm_generate
llm-docsum-tgi-server  |     response = llm_chain.invoke(input.query)
llm-docsum-tgi-server  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/langchain/chains/base.py", line 170, in invoke
llm-docsum-tgi-server  |     raise e
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/langchain/chains/base.py", line 160, in invoke
llm-docsum-tgi-server  |     self._call(inputs, run_manager=run_manager)
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/langchain/chains/combine_documents/base.py", line 138, in _call
llm-docsum-tgi-server  |     output, extra_return_dict = self.combine_docs(
llm-docsum-tgi-server  |                                 ^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/langchain/chains/combine_documents/map_reduce.py", line 232, in combine_docs
llm-docsum-tgi-server  |     [{self.document_variable_name: d.page_content, **kwargs} for d in docs],
llm-docsum-tgi-server  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-docsum-tgi-server  |   File "/home/user/.local/lib/python3.11/site-packages/langchain/chains/combine_documents/map_reduce.py", line 232, in <listcomp>
llm-docsum-tgi-server  |     [{self.document_variable_name: d.page_content, **kwargs} for d in docs],
llm-docsum-tgi-server  |                                    ^^^^^^^^^^^^^^
llm-docsum-tgi-server  | AttributeError: 'str' object has no attribute 'page_content'

Works fine when set "streaming": True

hteeyeoh commented 1 month ago

Changes merged into main branch. Closing this ticket