iMeanAI / WebCanvas

Connect agents to live web environments evaluation.
https://www.imean.ai/web-canvas
MIT License
180 stars 9 forks source link

Calculation Error in Token Cost Accumulation #25

Closed ZeonLap closed 1 week ago

ZeonLap commented 1 week ago

Thanks for the great work!

I've noticed an issue in the function save_token_count_to_file where the token cost calculation logic seems to have an error. Specifically, the total_input_token_cost and total_output_token_cost are repeatedly accumulated.

The problematic part of the code is here:

data["total_input_token_cost"] += data["total_planning_input_token_cost"] + data["total_reward_input_token_cost"]
data["total_output_token_cost"] += data["total_planning_output_token_cost"] + data["total_reward_output_token_cost"]
data["total_token_cost"] += data["total_input_token_cost"] + data["total_output_token_cost"]

Suggested Fix:

data["total_input_token_cost"] = data["total_planning_input_token_cost"] + data["total_reward_input_token_cost"]
data["total_output_token_cost"] = data["total_planning_output_token_cost"] + data["total_reward_output_token_cost"]
data["total_token_cost"] = data["total_input_token_cost"] + data["total_output_token_cost"]

Thanks for your work on this project!

han032206 commented 1 week ago

Thanks for bringing this up and for your feedback!

Our current implementation logic is designed to log the token consumption in real-time with a write and read operation per task, accumulating the data until the final calculation is completed. We’ll review this part for any possible improvements.