PEP 750: Tag Strings For Writing Domain-Specific Languages

[jimbaker]

This PEP introduces tag strings for custom, repeatable string processing. Tag strings are an extension to f-strings, with a custom function -- the "tag" -- in place of the f prefix. This function can then provide rich features such as safety checks, lazy evaluation, domain specific languages (DSLs) for web templating, and more.

Tag strings are similar to JavaScript tagged template literals and related ideas in other languages.

---

📚 Documentation preview 📚: https://pep-previews--3858.org.readthedocs.build/

[cpython-cla-bot[bot]]

All commit authors signed the Contributor License Agreement.

[lysnikolaou]

@python/pep-editors Can we please get another review on this?

[Carreau]

This is great. I hope we'll get log.debug"", log.info"" and alike. I've so many project use fstrings in logging, which prevents doing proper structured logging later. Though I do have the impression that tags having side-effects is not mentioned in the pep (beyond memoizing), but most example and discussions are around pure tags I think.

I'm also guessing projects like mypy will automatically detect (*args: Decoded | Interpolation) -> Any as potential tags.

[jimbaker]

This is great. I hope we'll get log.debug"", log.info"" and alike.

In this PEP, we don't support dotted names as prefixes for tag strings. We looked into it, and it required a fairly large number of parser changes. We decided to keep it simple instead. But dotted names should be supportable in a future PEP, if there's interest.

I've so many project use fstrings in logging, which prevents doing proper structured logging later. Though I do have the impression that tags having side-effects is not mentioned in the pep (beyond memoizing), but most example and discussions are around pure tags I think.

So the way I would think we would support logging integration is as follows, and it would enable structured logging:

Tag function. Create a tag function struct_log (it could be imported with a short name like sl). It has a signature like so:

def struct_log(*args: str | Interpolation) -> LogRecordMessage:
    ...

where LogRecordMessage holds the strings and interpolations for evaluation, if and when written to a log. It should be a simple wrapper, and is used to support mixed usage with the usual strings and args passed into log.info, etc.

@dataclass
class LogRecordMessage:
    args: tuple[str | Interpolation, ...]

A few things:

• Logging most likely doesn't need Decoded.raw, so str can be used.
• The specific tag function could be built by some sort of configurator. Example: provide a tag function that expands all usages of {expr} to {expr=value}.
• Likewise, this can be configured such that structured logging can use Interpolation.expr when writing out a JSON structured log.
• Another possible configuration would be Interpolation.getvalue when evaluated returns a Pydantic model, it can use that library's support for serializing to JSON. Lots of possibilities.

Now add the following to the above class:

    def __str__(self):
         # evaluate as if a lazy f-string

With __str__, this provides instant integration with logging given that LogRecord.getMessage calls str on the message.

Logging integration. The most straightforward way to do this is with setLogRecordFactory on a logger; this factory will be a subclass of LogRecord; it will have its own getMessage. This then works with a subclass of Formatter to build the final text representation, whether that's output of the above as if it were a f-string; or a JSON structured log version. I haven't used this library, but https://github.com/madzak/python-json-logger is one possible example for supporting such structured logging. There are also examples in the Logging Cookbook.

I'm also guessing projects like mypy will automatically detect (*args: Decoded | Interpolation) -> Any as potential tags.

That's an interesting idea. Definitely it goes the opposite way.

[jimbaker]

This PEP should be ready for approval, now that all earlier questions have been addressed. @python/pep-editors

[lysnikolaou]

Thanks everyone for the reviews! I'll merge this to get the conversation started and we can always come back to fix more issues.