search

spcl / graph-of-thoughts

Official Implementation of "Graph of Thoughts: Solving Elaborate Problems with Large Language Models"
https://arxiv.org/pdf/2308.09687.pdf
Other
2.1k stars 152 forks source link

Too many tokens #19

Closed cxxszz closed 10 months ago

cxxszz commented 10 months ago

Dear authors,

When I run the example in Quick Start, I have the following error:

Traceback (most recent call last):
  File "/home/weijie/graph-of-thoughts/graph_of_thoughts/quick_start.py", line 32, in <module>
    ctrl.run()
  File "/home/weijie/graph-of-thoughts/graph_of_thoughts/controller/controller.py", line 77, in run
    current_operation.execute(
  File "/home/weijie/graph-of-thoughts/graph_of_thoughts/operations/operations.py", line 120, in execute
    self._execute(lm, prompter, parser, **kwargs)
  File "/home/weijie/graph-of-thoughts/graph_of_thoughts/operations/operations.py", line 453, in _execute
    lm.query(prompt, num_responses=self.num_branches_response)
  File "/home/weijie/graph-of-thoughts/graph_of_thoughts/language_models/chatgpt.py", line 79, in query
    response = self.chat([{"role": "user", "content": query}], num_responses)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/weijie/miniconda3/envs/aca/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/weijie/graph-of-thoughts/graph_of_thoughts/language_models/chatgpt.py", line 121, in chat
    response = openai.ChatCompletion.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/weijie/miniconda3/envs/aca/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/weijie/miniconda3/envs/aca/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 153, in create
    response, _, api_key = requestor.request(
                           ^^^^^^^^^^^^^^^^^^
  File "/home/weijie/miniconda3/envs/aca/lib/python3.11/site-packages/openai/api_requestor.py", line 298, in request
    resp, got_stream = self._interpret_response(result, stream)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/weijie/miniconda3/envs/aca/lib/python3.11/site-packages/openai/api_requestor.py", line 700, in _interpret_response
    self._interpret_response_line(
  File "/home/weijie/miniconda3/envs/aca/lib/python3.11/site-packages/openai/api_requestor.py", line 765, in _interpret_response_line
    raise self.handle_error_response(
openai.error.InvalidRequestError: This model's maximum context length is 4097 tokens. However, you requested 5715 tokens (1619 in the messages, 4096 in the completion). Please reduce the length of the messages or completion.

Note that I saved the example as quick_start.py. How should this be fixed?

Thank you for your time and consideration.

Best regards, Weijie Liu

nblach commented 10 months ago

This is due to the max_tokens flag, which is provided via the config file and passed to the ChatCompletion request. The max_tokens flag determines the maximum number of tokens that the model can generate as part of the completion. However, the number of tokens in the prompt + max_tokens must be <= the context size of the model you are using.

The fix is to either adjust the value of the max_tokens flag or removing it entirely from the ChatCompletion request (then the model will just use the remaining number of tokens in the context as the limit).

Hope this helps! If not, please reopen the issue :)

-- Nils