Open Grollicus opened 11 months ago
The following code seems to be working and not generating Traceback. It's using the latest Jinja2 v3.1.4
from jinja2 import Environment, FileSystemLoader, ext
from jinja2.ext import Extension
from jinja2.lexer import Token, TOKEN_VARIABLE_BEGIN, TOKEN_VARIABLE_END
class LatexEscapeExtension(Extension):
def filter_stream(self, stream):
for token in stream:
if token.type == TOKEN_VARIABLE_BEGIN:
yield token
yield Token(token.lineno, 'lparen', '(')
elif token.type == TOKEN_VARIABLE_END:
yield Token(token.lineno, 'rparen', ')')
yield Token(token.lineno, 'pipe', '|')
yield Token(token.lineno, 'name', 'escape_tex')
yield token
else:
yield token
# Define the LaTeX escaping filter
def escape_tex(value):
escape_map = {
'&': r'\&',
'%': r'\%',
'$': r'\$',
'#': r'\#',
'_': r'\_',
'{': r'\{',
'}': r'\}',
'~': r'\textasciitilde{}',
'^': r'\textasciicircum{}',
'\\': r'\textbackslash{}',
}
return ''.join(escape_map.get(char, char) for char in value)
# Set up the Jinja environment and register the custom extension and filter
env = Environment(loader=FileSystemLoader('templates'), extensions=[LatexEscapeExtension])
env.filters['escape_tex'] = escape_tex
# Sample template string with LaTeX special characters
template_str = """
This is a LaTeX document with a variable: {{ variable }}
"""
# Create the template
template = env.from_string(template_str)
# Render the template with a sample value containing LaTeX special characters
output = template.render(variable="100% safe & secure $5 deal with #1 quality")
print(output)
Note that in my class above if you replace ==
to is
like in the original code, the same expected token 'end of print statement', got ')'
will be returned.
But from the documentation it reads The type of the token. This string is interned so you may compare it with arbitrary strings using the is operator.
adding @Tyl13 for further analysis.
It seems that the memory address of the enums are not actually being saved when it comes to when the input is tokenized. I'm not sure where the issue is coming from as the intern should be forcing it to be saved in the same memory space. Could be an issue coming internally of Python, but I'm not sure.
I'm using an extension modifying the token stream via
filter_stream
to implement autoescape functionality for a latex renderer.Extension looks like this:
It pipes every variable through an
escape_tex
-function, for example{{ thingy }}
gets translated to{{ (thingy) | escape_tex }}
.With
jinja2==2.11.3
this works as intended. With newer versions, for examplejinja2==3.1.2
I get this error:The full exception when disabling
rewrite_traceback_stack
indicates it crashes inparser.py:subparse
.On the other hand, directly parsing
env.from_string("{{ (foo) | escape_tex }}")
works and results in the same token stream, which to me indicates that something in the parsing decisions must go differently to lead to the exception.I'm unsure if this usage of
filter_stream
is not supported but the gettext example also rewrites the token stream. If you don't consider this a bug I'd of couse love other suggestions to work around this.Environment: