search

langchain-ai / langserve

LangServe 🦜️🏓
Other
1.94k stars 217 forks source link

Check how to get token counts from the remote client #283

Open eyurtsev opened 11 months ago

eyurtsev commented 11 months ago

Thx, @Eugene Yurtsev . I tried setting "include_callback_events=True" but to no avail. I simply wasn't able to programmatically track the token spend with "get_openai_callback()" when running the chain as a RemoteRunnable. When running a chain directly (i.e. not as a remote runnable), then the "get_openai_callback()" works well and picks up token spent and cost only when using the invoke method and having the "streaming=False" on the LLM side. When using the stream method,. then it does not work. Is there maybe an example our there for tracking the token spent and costs with a RemoteRunnable? Would be great if that could be resolved (server side OK as well), as hard to go to production without it. Thx

https://discord.com/channels/1038097195422978059/1170024642245832774/1180560916601901056

DMLON commented 9 months ago

Adding the OpenAICallbackHandler as a callback and setting include_callback_events to True makes pydantic go crazy and spew lots of errors. I've wasted like 5 hours trying to see token usage using langserve, still can't find a way Tried as well adding in the source code the actual context manager with get_openai_callback and same issue arises

Traceback (most recent call last):

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 404, in run_asgi

    result = await app(  # type: ignore[func-returns-value]

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__

    return await self.app(scope, receive, send)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\fastapi\applications.py", line 1054, in __call__

    await super().__call__(scope, receive, send)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\applications.py", line 123, in __call__

    await self.middleware_stack(scope, receive, send)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\middleware\errors.py", line 186, in __call__

    raise exc

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\middleware\errors.py", line 164, in __call__

    await self.app(scope, receive, _send)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\middleware\exceptions.py", line 62, in __call__

    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app

    raise exc

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app

    await app(scope, receive, sender)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\routing.py", line 762, in __call__

    await self.middleware_stack(scope, receive, send)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\routing.py", line 782, in app

    await route.handle(scope, receive, send)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\routing.py", line 297, in handle

    await self.app(scope, receive, send)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\routing.py", line 77, 

in app

    await wrap_app_handling_exceptions(app, request)(scope, receive, send)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app

    raise exc

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app

    await app(scope, receive, sender)

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\starlette\routing.py", line 72, 

in app

    response = await func(request)

               ^^^^^^^^^^^^^^^^^^^

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\fastapi\routing.py", line 299, in app

    raise e

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\fastapi\routing.py", line 294, in app

    raw_response = await run_endpoint_function(

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\fastapi\routing.py", line 191, in run_endpoint_function

    return await dependant.call(**values)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\langserve\server.py", line 465, 

in invoke

    result = await api_handler.invoke(request)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\langserve\api_handler.py", line 

719, in invoke

    self._InvokeResponse(

  File "F:\Proyectos\GPT\RAG\langchain_rag\Lib\site-packages\pydantic\v1\main.py", line 341, 

in __init__

    raise validation_error

pydantic.v1.error_wrappers.ValidationError: 27 validation errors for agentInvokeResponse     

callback_events -> 15 -> __root__ -> serialized

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_chain_start' (type=value_error.const; given=on_agent_finish; permitted=('on_chain_start',))

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_chain_end' (type=value_error.const; given=on_agent_finish; permitted=('on_chain_end',))

callback_events -> 15 -> __root__ -> error

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_chain_error' (type=value_error.const; given=on_agent_finish; permitted=('on_chain_error',))

callback_events -> 15 -> __root__ -> serialized

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> messages

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_chat_model_start' (type=value_error.const; given=on_agent_finish; permitted=('on_chat_model_start',))

callback_events -> 15 -> __root__ -> serialized

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> prompts

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_llm_start' (type=value_error.const; given=on_agent_finish; permitted=('on_llm_start',))

callback_events -> 15 -> __root__ -> response

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_llm_end' (type=value_error.const; given=on_agent_finish; permitted=('on_llm_end',))

callback_events -> 15 -> __root__ -> serialized

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> input_str

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_tool_start' (type=value_error.const; given=on_agent_finish; permitted=('on_tool_start',))

callback_events -> 15 -> __root__ -> output

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_tool_end' (type=value_error.const; given=on_agent_finish; 

permitted=('on_tool_end',))

callback_events -> 15 -> __root__ -> error

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_tool_error' (type=value_error.const; given=on_agent_finish; permitted=('on_tool_error',))

callback_events -> 15 -> __root__ -> serialized

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> query

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_retriever_start' (type=value_error.const; given=on_agent_finish; permitted=('on_retriever_start',))

callback_events -> 15 -> __root__ -> documents

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_retriever_end' (type=value_error.const; given=on_agent_finish; permitted=('on_retriever_end',))

callback_events -> 15 -> __root__ -> error

  field required (type=value_error.missing)

callback_events -> 15 -> __root__ -> type

  unexpected value; permitted: 'on_retriever_error' (type=value_error.const; given=on_agent_finish; permitted=('on_retriever_error',))
yj-ang commented 8 months ago

I'm also getting the same Pydantic error as above when setting include_callback_events=True. 🤔