KeyError: 'position_ids' when using facebook/opt-350m

[jvhoffbauer]

The bug I get a KeyError: 'position_ids' when using facebook/opt-350m which is a LLM available via the Huggingface model hub.

To Reproduce Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.

import guidance

llm = guidance.llms.Transformers("facebook/opt-350m")
guidance.llms.Transformers.cache.clear()

program = guidance("""Tweak this proverb to apply to model instructions instead.

{{proverb}}
- {{book}} {{chapter}}:{{verse}}

UPDATED
Where there is no guidance{{gen 'rewrite' stop="- "}}
- GPT {{gen 'chapter' max_tokens=10 pattern="[0-9]+"}}:{{gen 'verse' stop='\\n'}}""", llm=llm)

program(
    proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
    book="Proverbs",
    chapter=11,
    verse=14
)

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Ubuntu
• Guidance Version (guidance.__version__): 0.0.55

The full error is

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/guidance/_program_executor.py", line 94, in run
    await self.visit(self.parse_tree)
  File "/usr/local/lib/python3.9/dist-packages/guidance/_program_executor.py", line 429, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/usr/local/lib/python3.9/dist-packages/guidance/_program_executor.py", line 429, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/usr/local/lib/python3.9/dist-packages/guidance/_program_executor.py", line 218, in visit
    visited_children = [await self.visit(child, next_node, next_next_node, prev_node, node, parent_node) for child in node.children]
  File "/usr/local/lib/python3.9/dist-packages/guidance/_program_executor.py", line 218, in <listcomp>
    visited_children = [await self.visit(child, next_node, next_next_node, prev_node, node, parent_node) for child in node.children]
  File "/usr/local/lib/python3.9/dist-packages/guidance/_program_executor.py", line 429, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/usr/local/lib/python3.9/dist-packages/guidance/_program_executor.py", line 292, in visit
    command_output = await command_function(*positional_args, **named_args)
  File "/usr/local/lib/python3.9/dist-packages/guidance/library/_gen.py", line 132, in gen
    gen_obj = await parser.llm_session(
  File "/usr/local/lib/python3.9/dist-packages/guidance/llms/_transformers.py", line 368, in __call__
    generated_sequence = self.llm._generate_call(**generate_args)
  File "/usr/local/lib/python3.9/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/transformers/generation/utils.py", line 1391, in generate
    return self.greedy_search(
  File "/usr/local/lib/python3.9/dist-packages/transformers/generation/utils.py", line 2176, in greedy_search
    model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/guidance/llms/_transformers.py", line 185, in decorate_prep_step
    model_kwargs["position_ids"] = model_kwargs["position_ids"][:,len(self._prefix_cache):] # and update position ids
KeyError: 'position_ids'

Error in program:  'position_ids'

[slundberg]

Thanks! OPT handles the position ID alignment with the KV cache internally, so we can just skip that step if there are no position ids. Just pushed a fix.

[cpcdoy]

Reopening as a similar issue is back in 0.1.10:

Code:

from guidance import models, select

gpt = models.Transformers('facebook/opt-125m')

prompt = "X"
labels = ['a', 'b']

gpt + prompt + select(labels) 

Output: TypeError: OPTForCausalLM.forward() got an unexpected keyword argument 'position_ids'.

@slundberg Let me know if I can help by making a PR or if this can be quickly fixed on your end

[zzdhybthu]

It seems bug still exists?