Maximum recursion depth exceeded in comparison

[seanlynch]

The bug Occasionally I'm getting "maximum recursion depth exceeded" when attempting to generate results.

Traceback (most recent call last):
  File "/home/seanl/project/holodeck/holodeck/repro.py", line 20, in <module>
    main()
  File "/home/seanl/project/holodeck/holodeck/repro.py", line 12, in main
    llm
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/models/_model.py", line 204, in __add__
    return lm.run_stateless(value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/models/_model.py", line 303, in run_stateless
    for new_bytes, is_generated, new_bytes_log_prob, capture_groups, capture_group_log_probs, new_token_count in gen_obj:
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/models/_local.py", line 357, in __call__
    parse_tree = parser.parse_tree()
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/_parser.py", line 395, in parse_tree
    self._compute_parse_tree(0, root_item, reversed_state_sets)
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  [Previous line repeated 988 more times]
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/_parser.py", line 401, in _compute_parse_tree
    assert self._compute_children(pos, item, reversed_state_sets)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/_parser.py", line 443, in _compute_children
    if self._compute_children(inner_item.start, item, reversed_state_sets, values_pos + 1):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seanl/project/holodeck/venv/lib/python3.11/site-packages/guidance/_parser.py", line 424, in _compute_children
    return state_set_pos == item.start # note that ".start" mean end because items are reversed
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded in comparison

To Reproduce Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using. I've seen this with multiple models now, and it happens with Transformers and with LlamaCpp and LlamaCppChat.

from guidance import models, gen

def main():
    llm = models.LlamaCpp(
        "/home/seanl/SD/text-generation-webui/models/augmental-13b.Q5_K_M.gguf",
        n_gpu_layers=-1,
    )

    while True:
        lm = (
            llm
            + "Pick a number between 1 and 10: "
            + gen(name="guess", stop="\n", temperature=0.8)
        )
        print(lm["guess"])

if __name__ == "__main__":
    main()

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Pop_OS (Ubuntu derivative)
• Guidance Version (guidance.__version__): 0.1.1

[baleksey]

Have the same problem. Also on ver: 0.1.1, Ubuntu, Using LLamaCPP

>> Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/root/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/root/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/root/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/root/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/root/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/app/libs/guidance_alx/guide_lcpp.py", line 183, in <module>
    out = GD.run(prompt, max_tokens=250, temperature=0.30, top_p=1.0, );print(f"\n\n") #stop=[".","!","?","\n"]
  File "/app/libs/guidance_alx/guide_lcpp.py", line 90, in run
    self.lm = self.model + prompt
  File "/usr/local/lib/python3.10/dist-packages/guidance/models/_model.py", line 204, in __add__
    return lm.run_stateless(value)
  File "/usr/local/lib/python3.10/dist-packages/guidance/models/_model.py", line 303, in run_stateless
    for new_bytes, is_generated, new_bytes_log_prob, capture_groups, capture_group_log_probs, new_token_count in gen_obj:
  File "/usr/local/lib/python3.10/dist-packages/guidance/models/_local.py", line 357, in __call__
    parse_tree = parser.parse_tree()
  File "/usr/local/lib/python3.10/dist-packages/guidance/_parser.py", line 395, in parse_tree
    self._compute_parse_tree(0, root_item, reversed_state_sets)
  File "/usr/local/lib/python3.10/dist-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  File "/usr/local/lib/python3.10/dist-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  File "/usr/local/lib/python3.10/dist-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  [Previous line repeated 977 more times]
  File "/usr/local/lib/python3.10/dist-packages/guidance/_parser.py", line 401, in _compute_parse_tree
    assert self._compute_children(pos, item, reversed_state_sets)
  File "/usr/local/lib/python3.10/dist-packages/guidance/_parser.py", line 417, in _compute_children
    item.children = [None for _ in range(len(item.values))]
  File "/usr/local/lib/python3.10/dist-packages/guidance/_parser.py", line 417, in <listcomp>
    item.children = [None for _ in range(len(item.values))]
  File "_pydevd_bundle/pydevd_cython.pyx", line 1457, in _pydevd_bundle.pydevd_cython.SafeCallWrapper.__call__
RecursionError: maximum recursion depth exceeded while calling a Python object

[baleksey]

I've found a temporary solution. In my case it always shows an error when generation is little over max_tokens set (or default gen value). For example, max_tokens=400 and it fires on 415 total tokens generated. So it somehow doesn't stop on max token reached and fire this RecursionError in parser.

Temp solution: always set bigger (or max possible) max_tokens value for gen() function and assign proper stop criteria in order to stop generation

Update: Nope, not always a solution.

[hcombz]

I ran into the same problem using guidance==0.1.1 and openai==1.3.1.

Here is the code

from guidance import models, user, assistant, gen, system

gpt = models.OpenAI("gpt-3.5-turbo")
with system():
    lm = gpt + "You are a recruiter writing a job offer for a car maker company"

    lm += "Here are the informations you have about the job opening.\n\
    - Job title : Mechanic\n\
    - Location: Seattle\n"

with user():
    lm += "Can tou write the job offer section that details the tasks the employee wil have to handle?"

print(lm)
with assistant():
    lm += gen("desc_ent", max_tokens=400)
print(lm)

Here is the output

<|im_start|>system
You are a recruiter writing a job offer for a car maker companyHere are the informations you have about the job opening.
    - Job title : Mechanic
    - Location: Seattle
<|im_end|><|im_start|>user
Can tou write the job offer section that details the tasks the employee wil have to handle?<|im_end|>
Traceback (most recent call last):
  File "/home/hcombaluzier/Documents/Project_group/CRIT/redaction-automatique-offre/test.py", line 29, in <module>
    lm += gen("desc_ent", max_tokens=400)
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/models/_model.py", line 204, in __add__
    return lm.run_stateless(value)
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/models/_model.py", line 303, in run_stateless
    for new_bytes, is_generated, new_bytes_log_prob, capture_groups, capture_group_log_probs, new_token_count in gen_obj:
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/models/_local.py", line 357, in __call__
    parse_tree = parser.parse_tree()
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/_parser.py", line 395, in parse_tree
    self._compute_parse_tree(0, root_item, reversed_state_sets)
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/_parser.py", line 410, in _compute_parse_tree
    self._compute_parse_tree(pos, child, reversed_state_sets)
  [Previous line repeated 989 more times]
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/_parser.py", line 401, in _compute_parse_tree
    assert self._compute_children(pos, item, reversed_state_sets)
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/guidance/_parser.py", line 439, in _compute_children
    for inner_item in reversed_state_sets[state_set_pos]:
  File "/home/hcombaluzier/miniconda3/envs/crit/lib/python3.10/site-packages/ordered_set/__init__.py", line 310, in __iter__
    return iter(self.items)
RecursionError: maximum recursion depth exceeded while calling a Python object

I also have some (relatated?) issues with truncated ouptuts since i updated to 0.1.1 (Sorry but i was unable to reproduce this within a minimal english example). Here the generated output is stopped in the middle of a phrase. When using the direct openai api with the same prompt I get a complete output. <|im_start|>user\n\nVoici les informations relatives à l'offre\n- Intitulé de l'offre: Agent de fabrication\n- Lieu: Batilly\n- Durée de contrat: 1 semaine à 18 mois\n- Horaires: 35 heures ou 2x8\n- Modalité de salaire: horaire\n- Salaire: 13€\n- entreprise: Renault\n- Expérience souhaitée: 0-1 an\n- Etude: BEP/CAP\nLa section description de l'entreprise contient une description générique de Renaultrédigée d'un point de vue externe. \n Il ne faut pas écrire nous ni notre.\nRédige la section.\n<|im_end|><|im_start|>assistant\nNous recherchons actuellement un agent de fabrication pour notre entreprise située à Batilly. En tant qu'agent de fabrication, vous serez responsable de la production et de l'assemblage de nos produits automobiles. \n\nVotre mission consistera à effectuer différentes tâches telles que l'assemblage de pièces, le contrôle qualité, la maintenance de premier niveau et le respect des normes de sécurité. Vous travaillerez en équipe et serez<|im_end|

Thank you for your help

[hcombz]

I was able to run my example installing guidance from source, tracking errors and making some modifications to the source code.

• The first modification I had to make was to add a regex to the gen function, if not the zero_or_more function adds some grammar components that leads to the maximum recursion issue

The second problem I encountered was dealing with max_tokens.

• I do not understand why the attribute max_token is forced to 100 in the guidance/guidance/models/_openai.py(l.87)
• The max_token argument is also not well forwarded to the run_stateless function, max_tokens keeps the default value = 1000 (guidance/guidance/models/_model.py (l.279))

In my experiences, all those values should equals to NUMBER_OF_TOKENS when calling gen(names, max_tokens=NUMBER_OF_TOKENS) When using a high number of tokens I had to also modify the value of the max_calls attribute that is set in guidance/guidance/models/_remote.py (l.90)

Here is the functional version of my code

gpt = models.OpenAI("gpt-3.5-turbo")
with system():
    lm = gpt + "You are a recruiter writing a job offer for a car maker company"

    lm += "Here are the informations you have about the job opening.\n\
    - Job title : Mechanic\n\
    - Location: Seattle\n"

with user():
    lm += "Can tou write the job offer section that details the tasks the employee wil have to handle?"

print(lm)
with assistant():
    lm += gen("desc_ent", max_tokens=400, regex="(.)*") #the regex does not constrained the ouput
print(lm)

Note that I had to set max_calls=100 and set the default values in _openai.py and _model.py to 400

I am not sure this is the proper way to use guidance. I do not understand everything about how the streaming mode works and I won't be able to make a valid fix myself but it would be great if someone maintaining the repo could take a look at this.

Thanks

[seanlynch]

The second problem I encountered was dealing with max_tokens.

* I do not understand why the attribute max_token is forced to 100 in the `guidance/guidance/models/_openai.py `(l.87)

Could this be the cause of #451 ?

[slundberg]

Thanks for highlighting this! I just converted a couple functions from recursive to iterative formulations, and that fixed the issue as I was able to reproduce it. Please let us know if you run into more recursion errors.