search

billxbf / ReWOO

Decoupling Reasoning from Observations for Efficient Augmented Language Models
MIT License
858 stars 68 forks source link

Token price human error issue #13

Open naem1023 opened 1 month ago

naem1023 commented 1 month ago

Hi, I read the paper ReWOO very impressive.

I think I found some human error when counting the token cost on the PWS.py https://github.com/billxbf/ReWOO/blob/9cd0283043ff4be0c9d614fda2789d143ca6ffd1/algos/PWS.py#L66-L68

OpenAI count the input token and output token because they have different pricing policy, but these codes just count the input and output token as same. Maybe I can make a mistake, but I want you to know the issue.

Thanks a lot for the nice research!!

billxbf commented 1 month ago

Thanks for your kind words! You are right, this function doesn't treat in/out token prices separately. If I remember correctly, openai's API didn't split the in/out token prices at the time of this work. So I'd rather treat it as an estimation for experiments at the time. Now they include token counts in the API's completion object, accessed by completion["usage"]

naem1023 commented 1 month ago

Because of your response, now I totally understand about the history of the works. Thanks a lot!!