feat: record user token usage

[RaoHai]

• 记录用户 / bot 的 token 用量

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
petercat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 11, 2024 9:04am

[petercat-assistant[bot]]

Walkthrough

此 PR 主要增加了记录用户和 bot 的 token 用量的功能。通过在多个模块中添加新的表和字段来实现 token 用量的记录。

Changes

文件路径	概述
migrations/supabase/migrations/20240905155017_remote_schema.sql	添加了 domain_whitelist 和 repo_name 字段到 bots 表，添加了 page_index 字段到 git_issue_tasks 表
migrations/supabase/migrations/20240909061136_remote_schema.sql	创建了 user_llm_tokens 表，并在多个表中添加了 encrypted_token 和 user_id 字段
migrations/supabase/migrations/20240909084251_remote_schema.sql	在 user_llm_tokens 表中添加了 sanitized_token 字段
migrations/supabase/migrations/20240911082857_remote_schema.sql	创建了 user_token_usage 表，并在多个表中添加了新的字段和索引
server/agent/base.py	添加了 dict_to_sse 函数，修改了 run_stream_chat 方法以返回字典而不是字符串
server/agent/bot/init.py	添加了 token_id 属性
server/agent/bot/bot_builder.py	修改了 run_stream_chat 方法以使用 dict_to_sse
server/agent/qa_chat.py	修改了 agent_stream_chat 方法以返回字典而不是字符串
server/chat/router.py	添加了 create_token_recorder 函数，并修改了 agent_stream_chat 的调用管道
server/core/dao/userTokenUsageDAO.py	新增了 UserTokenUsageDAO 类，用于操作 user_token_usage 表
server/core/models/user_token_usage.py	新增了 UserTokenUsage 模型
server/core/service/user_token_usage.py	新增了 create_token_recorder 函数，用于记录 token 用量
server/requirements.txt	添加了 toolz 依赖

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 57.14286% with 33 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
server/core/service/user_token_usage.py	50.00%	13 Missing :warning:
server/agent/base.py	20.00%	12 Missing :warning:
server/chat/router.py	50.00%	3 Missing :warning:
server/core/dao/userTokenUsageDAO.py	72.72%	3 Missing :warning:
server/agent/bot/__init__.py	66.66%	1 Missing :warning:
server/agent/bot/bot_builder.py	50.00%	1 Missing :warning:

Files with missing lines	Coverage Δ
server/agent/qa_chat.py	47.36% <100.00%> (ø)
server/core/models/user_token_usage.py	100.00% <100.00%> (ø)
server/agent/bot/__init__.py	68.00% <66.66%> (-0.19%)	:arrow_down:
server/agent/bot/bot_builder.py	72.72% <50.00%> (ø)
server/chat/router.py	59.45% <50.00%> (+0.08%)	:arrow_up:
server/core/dao/userTokenUsageDAO.py	72.72% <72.72%> (ø)
server/agent/base.py	24.29% <20.00%> (-1.22%)	:arrow_down:
server/core/service/user_token_usage.py	50.00% <50.00%> (ø)

[RaoHai]

@xingwanying 写好了老师