Can Telemetry tracking be used when using TaskWeaver as a library?

[uebhh]

Accessing taskweaver through the WebUI, tracking can be done through Telemetry, but when using TaskWeaver as a library, no tracking information is seen in Telemetry. Is it possible to perform tracking in this scenario, and how should it be used?

Environment Information (please complete the following information):

OS: Windows Python Version 3.11 LLM that you're using: GLM4

[liqul]

The tracing module is not tied to the WebUI, and therefore, you should be able to collect all the traces using TaskWeaver as a lib. When you say that you can see the information in "Telemetry", do you mean the Jaeger frontend? Could you shall more details on how you set up the tracing infrastructure and what are the configurations in the project config file?

[uebhh]

The tracing module is not tied to the WebUI, and therefore, you should be able to collect all the traces using TaskWeaver as a lib. When you say that you can see the information in "Telemetry", do you mean the Jaeger frontend? Could you shall more details on how you set up the tracing infrastructure and what are the configurations in the project config file?

The issue has been identified. The virtual environment for executing [Chainlit run app.py] is different from the one for executing [TaskWeaver as a lib]. After installing the OpenTelemetry library in the virtual environment where [TaskWeaver as a lib] is executed, it worked.

Through tracing, I found that the LLM interaction content is the same during the WebUI call and the [taskweaver as lib call]. Is the process: codeInterpreter(result) -> planner -> LLM -> user? It takes a long time to go through LLM again. Is it possible to improve the response speed when using [TaskWeaver as a lib] by following the sequence: codeInterpreter(result) -> planner -> caller?

[liqul]

The default flow is: caller -> planner -> codeinterpreter -> planner -> caller. By configuring, the flow could be changed to: caller -> codeinterpreter -> caller. If the result is passed to the planner, we need to call LLM to do a planning again to see if any actions need to take before responding to the caller (the same to the user case). For example, if the code interpreter failed, the planner may come up with a new plan to execute. That is why the planner needs to call LLM again when receiving the result from the code interpreter. But if you consider planner is not necessary in your scenario, you can configure the CodeInterpreter as the single role and bypass the planner.

[uebhh]

The default flow is: caller -> planner -> codeinterpreter -> planner -> caller. By configuring, the flow could be changed to: caller -> codeinterpreter -> caller. If the result is passed to the planner, we need to call LLM to do a planning again to see if any actions need to take before responding to the caller (the same to the user case). For example, if the code interpreter failed, the planner may come up with a new plan to execute. That is why the planner needs to call LLM again when receiving the result from the code interpreter. But if you consider planner is not necessary in your scenario, you can configure the CodeInterpreter as the single role and bypass the planner.

Thank you for your answer, it's very helpful. Are you suggesting that in order to avoid unnecessary interactions with the planner, can replace it with a development role to implement the functions within the CodeInterpreter ? However, I noticed that the development role mentioned in the documentation also needs to interact with the planner. How to configure CodeInterpreter as the single role and bypass the planner?

[liqul]

You can have the following configuration to enable the CodeInterpreter-only mode:

{
    "session.roles": ["code_interpreter"]
}

If it is not configured, the default is:

{
    "session.roles": ["planner", "code_interpreter"]
}

[uebhh]

Thank you, I'll give it a try!