Template with escaped vars fails

[tylerbrandt]

Summary

I was trying to include some JSON schema in my template and I ran across this bug where if the input contains an escaped identifier like $$var (per https://docs.python.org/3/library/string.html#template-strings), llm -t crashes.

Repro

1. Create template with escaped string

   llm --system 'Hello $$world' --save interpolation-bug

2. Invoke template (it doesn't matter what's in the prompt)

   llm -t interpolation-bug 'oops'
   Traceback (most recent call last):
   File "/Users/tyler.brandt/Library/Python/3.11/bin/llm", line 8, in <module>
   sys.exit(cli())
            ^^^^^
   File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 1157, in __call__
   return self.main(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 1078, in main
   rv = self.invoke(ctx)
        ^^^^^^^^^^^^^^^^
   File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 1688, in invoke
   return _process_result(sub_ctx.command.invoke(sub_ctx))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 1434, in invoke
   return ctx.invoke(self.callback, **ctx.params)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 783, in invoke
   return __callback(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/llm/cli.py", line 216, in prompt
   prompt, system = template_obj.evaluate(prompt, params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/llm/templates.py", line 31, in evaluate
   system = self.interpolate(self.system, params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/llm/templates.py", line 48, in interpolate
   "Missing variables: {}".format(", ".join(missing))
                                  ^^^^^^^^^^^^^^^^^^
   TypeError: sequence item 0: expected str instance, NoneType found

It looks like the error arises due to the use of template.pattern in extract_vars which returns a list of a single None value per escaped var; Template.get_identifiers (new in 3.11) does not exhibit this behavior (returns no list items instead for such vars).

[tylerbrandt]

For now I am working around it by using a var $schema and passing the value using -p schema "$(cat file)" but it would be nice to not have to do that.

[ryanhalliday]

Example command for what @tylerbrandt mentioned above:

llm -t your-template -p schema "$(cat your-actual-system-prompt.txt)"

Where your-tempalte.yaml should look something like this:

model: "gpt-4"
system: |
    $schema

Slightly annoying to work around, especially considering $$ escape doesn't work but it's good enough for now.

It does look like either of these PRs would resolve this from a quick look: https://github.com/simonw/llm/pull/469 or https://github.com/simonw/llm/pull/429 @simonw