AssertionError: Cross check last_pos in running notebooks/tutorials/intro_to_guidance.ipynb

[dercaft]

The bug Running notebooks/tutorials/intro_to_guidance.ipynb, I AssertionError. Markdown Cell:

## Generating lists

Whenever you want to generate a list of items you can use the `list_append` parameter which will cause the captured value to be appended to a list instead of overwriting previous values.

Code Cell:

lm = mistral + f'''\
Q: {query}
Now I will choose to either SEARCH the web or RESPOND.
Choice: {select(["SEARCH", "RESPOND"], name="choice")}
'''
if lm["choice"] == "SEARCH":
    lm += "Here are 3 search queries:\n"
    for i in range(3):
        lm += f'''{i+1}. "{gen(stop='"', name="queries", temperature=1.0, list_append=True)}"\n'''

Error is:

File <PATH>/guidance/guidance/models/_model.py:842, in Engine._cleanup_tokens(self, token_ids, token_byte_positions)
    839     # another ugly hack for tokenizers that are not stable on encode/decode cycles
    840     # currently only Phi-3, should generalize this method if we see more of these
    841     if not hasattr(self, "_disable_retokenize_check"):
--> 842         assert token_byte_positions[-1] == last_pos, "Cross check last_pos"
    844 return token_ids, token_byte_positions

AssertionError: Cross check last_pos

System info (please complete the following information):

• OS : Ubuntu 22.04
• Guidance Version (guidance.__version__):'0.1.14'

[derfred]

I get the same error when running with LllamaCpp backend on MacOS, with version 0.1.15

from guidance import models, gen, select llama2 = models.LlamaCpp("llama.cpp/Phi-3-medium-128k-instruct-Q5_K_M.gguf") llama2 + f'Do you want a joke or a poem? A {select(['joke', 'poem'])}.\nOkay, here is a one-liner: "{gen(stop='"')}"'

[fabriceyhc]

I am also getting the same error :(

import time
import guidance
from guidance import models, select

model_id = "TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ"
llm = models.Transformers(
    model_id, 
    device_map='auto'
)

@guidance
def annotate_drugabuse(lm, doc_id, sentence_id, medical_note):
    lm += f"""\
    ###Task Description: 
    Please carefully review the following medical note for any mentions of drug use.

    ###The medical note to evaluate:
    {medical_note}

    ###Feedback: 
    ```json
    {{
        "doc_id": "{doc_id}",
        "sentence_id": "{sentence_id}",
        "medical_note": "{medical_note}",
        "Heroin": {select(options=['True', 'False'], name='Heroin')},
        "Cocaine": {select(options=['True', 'False'], name='Cocaine')},
        "Methamphetamine": {select(options=['True', 'False'], name='Methamphetamine')},
        "Benzodiazepine": {select(options=['True', 'False'], name='Benzodiazepine')},
    }}```"""
    return lm

test = {
    "doc_id": 243,
    "sentence_id": 36, 
    "medical_note": "Patient reports diagnosis of bipolar disorder and polysubstance abuse in the past."
}

a = time.time()
lm = llm + annotate_drugabuse(**test)
print(lm)
time.time() - a

 '###Feedback: \n'
 '```json\n'
 '{\n'
 '    "doc_id": "243",\n'
 '    "sentence_id": "36",\n'
 '    "medical_note": "Patient reports diagnosis of bipolar disorder and '
 'polysubstance abuse in the past.",\n'
 '    "Heroin":<||_html:<span style=\'background-color: rgba(0.0, 165.0, 0, '
 "0.15); border-radius: 3px;' title='1.0'>_||> "
 '"<||_html:</span>_||><||_html:<span style=\'background-color: rgba(0.0, '
 "165.0, 0, 0.15); border-radius: 3px;' title='1.0'>_||>")
Traceback (most recent call last):
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data2/fabricehc/drugabuse/annotate.py", line 46, in <module>
    lm = llm + annotate_drugabuse(**test)
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/site-packages/guidance/models/_model.py", line 1163, in __add__
    out = value(lm)
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/site-packages/guidance/_grammar.py", line 69, in __call__
    return self.f(model, *self.args, **self.kwargs)
  File "/data2/fabricehc/drugabuse/annotate.py", line 10, in annotate_drugabuse
    lm += f"""\
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/site-packages/guidance/models/_model.py", line 1151, in __add__
    out = lm + partial_grammar
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/site-packages/guidance/models/_model.py", line 1159, in __add__
    out = lm._run_stateless(value)
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/site-packages/guidance/models/_model.py", line 1364, in _run_stateless
    for chunk in gen_obj:
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/site-packages/guidance/models/_model.py", line 736, in __call__
    is_done, logits_state, response_state = self.next(logits)
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/site-packages/guidance/models/_model.py", line 700, in next
    self._cleanup_tokens(
  File "/home/fabricehc/miniconda3/envs/da/lib/python3.10/site-packages/guidance/models/_model.py", line 842, in _cleanup_tokens
    assert token_byte_positions[-1] == last_pos, "Cross check last_pos"
AssertionError: Cross check last_pos

[alexandreteles]

Downgrading to llama-cpp-python==v0.2.75 should fix the issue until Guidance pushes an update.

[riedgar-ms]

We have changed the assert to a warning, since it's checking something which is not absolutely essential to guidance (although knowing that the check has failed is very useful, since it means that the tokeniser is doing something strange).

[dercaft]

Downgrading to llama-cpp-python==v0.2.75 should fix the issue until Guidance pushes an update.

It works! thank you