Create templates from non-Python files

[jimbaker]

So far we have been looking at templates defined in Python code. There are many advantages in doing so, including use of Python modules, imports, etc.

However, Jinja and similar systems do allow ready definition of templates from arbitrary files/sources. Python also allows code to be evaluated from arbitrarily sources. Is it possible to generate templates accordingly?

Here's a sketch of a possible solution. Let's start with this function:

def mytag(*args):
  return args

Create a template dynamically:

>>> x = eval('mytag"{alice} meet {bob}"')
>>> x
((<function <lambda> at 0x7f2886a734c0>, 'alice', None, None), ' meet ', (<function <lambda> at 0x7f2886a72d40>, 'bob', None, None))

Define a function to pass in context:

def g():
  alice = 'Alice'
  bob = 'Bob'
  return f'{x[0][0]()} meet {x[2][0]()}'

NOTE: the use of x[0][0]() etc in the line f'{x[0][0]()} meet {x[2][0]()}' is meant to represent compiled Python code, where x[0][0]() is an interpolation. So this is the equivalent of what is currently done in HTML compilation code.

And this will go boom:

>>> g()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in g
  File "<string>", line 1, in <lambda>
NameError: name 'alice' is not defined. Did you mean: 'slice'?

The problem here is that it's not possible as-is to use nested scopes with eval'ed code (https://docs.python.org/3/library/functions.html#eval). This is true even if we change g:

def g(alice, bob):
  return f'{x[0][0]()} meet {x[2][0]()}'

Same error results:

>>> g('Alice', 'Bob')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in g
  File "<string>", line 1, in <lambda>
NameError: name 'alice' is not defined. Did you mean: 'slice'?

But what if changed what we evaluated by knowing the variables alice and bob are required?

So let's try this code:

from typing import Any, Literal

def make_template(tagname, template):
    # Ideally we would pass in the tag function, not the tagname - this should
    # be possible

    # No support for template containing triple quotes, but this can be
    # addressed with suitable escaping

    def identity(*args):
        return args

    # should do a gensym here to ensure extract_tag, f are not in use
    namespace = {'extract_tag': identity, 'f': None}

    # do a first pass to extract any used names
    used_names = set()
    args = eval(f'extract_tag"""{template}"""', None, namespace)
    for arg in args:
        match arg:
           case getvalue, _, _, _:
              used_names |= set(getvalue.__code__.co_names)

    # then build a wrapper function for the template
    param_names = used_names - namespace.keys()
    params = ', '.join(sorted(param_names))
    code = f'def f({params}): return \\\n  {tagname}"""{template}"""'
    code_obj = compile(code, "<string>", "exec")
    exec(code_obj, None, namespace)
    return namespace['f']

def convert(value: Any, conv: Literal['r'] | Literal['s'] | Literal['a'] | None) -> str:
    match conv:
       case 's' | None:
          return str(value)
       case 'a':
          return ascii(value)
       case 'r':
          return repr(value)

def my_fstring(*args) -> str:
    x = []
    for arg in args:
        match arg:
            case str():
                x.append(arg)
            case getvalue, _, conv, format_spec:
                format_spec = '' if format_spec is None else format_spec
                x.append(convert(format(getvalue(), format_spec), conv))
    return ''.join(x)

template = make_template('my_fstring', '{alice} meet {bob}')
print(template(alice='Alice', bob='Bob'))

which works as expected.

[jimbaker]

In general, we don't want this in the PEP, because it relies on an implementation detail (Python code objects and their attributes); and it is nontrivial. But it's interesting that this is readily doable.

[jimbaker]

This could be a useful test for tag strings, at the very least, so I added that accordingly with https://github.com/pauleveritt/fdom/pull/20. I didn't do the last step of supporting gensym style symbols, but for testing that's not really necessary.