Closed ncoghlan closed 1 week ago
@nhumrich Since both our names are on the PEP, let me know if there's anything here that you'd prefer we didn't add (the lazy evaluation stuff in particular seems potentially controversial, although I must say I like the way that second logging example looks - just 4 extra characters to say "this is evaluated at rendering time, not template definition time")
Awesome. I will look more in depth the details required to add these things.
In the meantime, I am not convinced that lazy evaluation is something this PEP needs to deal with. It is not the purpose of the PEP to solve every possible case, but rather, allow the user to solve those problems in their own programs.
I think it's more likely that expensive_call
in a template literal string would be the format itself, rather than some arbitrary function wanting to be delayed.
Since the format function is already a callable, I think the normal use case of deferred execution is still easy and expected.
If for some reason, a user needed deferred execution of a function call inside the template itself, they could always add support for callables inside their own template renderer. Then use that in conjunction with lambdas or partials. Which, is no different from how you would handle it anywhere else in python.
Passing in your own custom format
function that handles callables feels so easy, I don't see why we would need to overcomplicate the PEP to handle it.
We could potentially add that delayed evaluation of the values themselves could be considered in a followup PEP if there warrants a need.
Your thoughts on declaring lazy evaluation out of scope make sense to me, so I've reworded the initial post accordingly (I also moved the suggested ()
for rendering time function invocation to the format specifier rather than having it outside the field specifier where render_field
implementations wouldn't be able to see it)
Edit: I also updated my notes on mentioning string.Template
Edit 2: added details on the changes to TemplateField
and TemplateLiteralField
to account for eager vs lazy evaluation
Added further notes on the potential utility for i18n use cases based on the PEP 750 discussion with @warsaw (short version: while some level of integration is theoretically possible, the potential benefits are small enough and the related challenges significant enough that it likely won't be worth spending anyone's time to actually make that integration happen).
This isn't in the main set of notes yet, but writing those up did give me an idea in relation to this comment @warsaw made in the PEP 750 thread:
Another thing that occurs to me though, is that to be a more-or-less drop in replacement for flufl.i18n style translatable strings, you’d need to be able to support the $-string (PEP 292) style placeholders. Is the PEP 750 placeholder syntax restricted to f-string style {placeholder} syntax?
It is specifically PEP 750 that commits all possible tagged strings to using f-string style interpolation fields, with no scope for future syntactic variations.
By contrast, PEP 501 only commits t-strings to directly aligning with the f-string syntax. That means it would leave the door open to a future PEP proposing a dedicated _
string prefix that instead used the PEP 292 syntax to define interpolation fields. I think that's a genuine design benefit arising from the more limited proposal, so it's likely worth mentioning.
Adding in another change inspired by PEP 750: making the interpretation of conversion specifiers lazy. While template literals will make a
, r
, and s
work as they do in f-strings by default, other template renderers may handle them differently. Template literals will also support ()
as a "call the result of the expression at rendering time" conversion specifier, giving a basic level of support for delayed rendering (enough for logging to support lazy field rendering without needing custom specifier processing).
I also added some further notes about the i18n use case, including the potential benefits of leaving the door open to a future syntactic proposal for "dollar-strings" (which would use the PEP 292 substitution syntax, but the same runtime interpolation machinery).
I'm going to start a draft PR for these changes tomorrow (Aug 25 AEST).
WIP PR on my fork of the PEPs repo: https://github.com/ncoghlan/peps/pull/8
( I won't merge it there, but this potentially allows comments on the amendments as they're in progress, rather than having to wait until I'm ready to submit the full PR to the main PEPs repo)
Decent progress today: WIP PR now has a first draft of the updated proposal and specification sections. I also added some notes on sections that should either be moved around, or else referenced earlier than they are.
Next step is to review the design discussion section and update it as needed.
First pass at updating the design discussion section (including adding a "How To Teach This" suggestion).
Still a few specific TODOs to fill in.
In filling out the design discussion section, I noticed a problem with the way custom conversions specifiers were defined: actually using them would break the default renderer. For now, I'm going to tweak the syntax to allow the default renderer to cleanly ignore them, but it may be simpler to only define the new lazy evaluation specifier. The reason I'm not going down the latter yet is because I have concerns that omitting custom conversion specifier support will lead to ad hoc conventions in the format specifier field that serve the same purpose as custom conversion specifiers.
OK, first pass at the update is done. Next steps before creating the PR against the PEPs repo:
Just noting an idea that I considered adding, but decided it didn't add enough to the PEP to be worth the extra up-front complexity (vs just adding it later as a regular feature request): a from_raw_segments
class method that handles wrapping plain strings and 4-tuples into TemplateLiteralText
and TemplateLiteralField
objects. The basic idea is simple enough, but the implementation gets fiddly if you want to make it do the right thing when the passed in values are already instances of those classes. It's also not clear yet if that's the right programmatic API to provide (for example, building on https://docs.python.org/3/library/string.html#string.Formatter is another potential candidate).
No ETA on a new discussion thread yet. There's heaps of time until Python 3.14's beta cycle starts, and I think discussions will be more productive if we wait until the next iteration of PEP 750 has been published so folks can more easily compare the latest versions of both proposals (they look more different than they actually are at the moment, since the PEP 501 update is based on the upcoming PEP 750 amendments that were already announced in the discussion thread rather than on the current version).
Accumulating ideas prompted by the PEP 750 discussion at https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408 before working on an update to the PEP 501 text:
format
builtin. This is an optional third argument toformat
that accepts the conversion specifier as a string. It defaults to the empty string (no conversion). Accepted values area
,r
, ands
(mapping to callingascii
,repr
, andstr
on the input as they do for f-strings), and()
(meaning "call the input value before formatting it`)render_field
to pass the conversion specifier as the third argument (using the empty string if no conversion specifier is present) instead of evaluating the conversion function eagerly. The addition to the signature offormat
means it remains usable as the default value forrender_field
.render_text
callback toTemplateLiteral.render
signature (default value:str
)typing.InterpolationTemplate
(protocol corresponding to the concretetypes.TemplateLiteral
type)typing.TemplateText
(equivalent toDecoded
from PEP 750)typing.TemplateField
(inspired byInterpolation
from PEP 750, with adjustments for eager field evaluation)types.TemplateLiteralText
(equivalent toDecodedConcrete
from PEP 750)types.TemplateLiteralField
(inspired byInterpolationConcrete
from PEP 750, with adjustments for eager field evaluation)TemplateLiteral
, producingTemplateLiteralText
andTemplateLiteralField
instances in their order of appearance (keeping the "no emptyTemplateLiteralText
entries" rule from PEP 750)TemplateLiteral
and implementation ofTemplateLiteral.render
in light of the above changes (in particular, consider switching it over to using PEP 750 style pattern matching)string.Template
and the i18n use case as per the notes below. In particular, note that the tagged string proposal effectively says "all compiler supported string interpolation must use f-string syntax", whereas PEP 501 just says "t-strings use the same interpolation syntax as f-strings", leaving the door open to other potential interpolation string syntaxes (such as dollar-strings).To adjust
TemplateField
for eager evaluation:getvalue
->value
(expression is eagerly evaluated at template definition time)conv
field (conversions are applied at template definition time)This gives the following interface for the concrete type:
Based on the discussions with @warsaw in the PEP 750 thread (e.g. https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408/122 and https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408/135 ), it's looking like neither template literals nor tagged strings would be particularly beneficial for i18n use cases.
It's definitely possible to integrate them:
string.Template
could support construction from the native template syntax (extracting the template's field names from the interpolation fields, together with a string-keyed dict mapping the field names to their eagerly interpolated values)string.Template
could implement the native template interpolation protocol, rendering itself in a normalised form (the simplest version would always render the fields as${name}
, but a slightly nicer version would emit$name
when it is unambiguous to do so)However, the integration would have significant caveats:
${...}
substitution form (since the compiler wouldn’t see$...
as defining an interpolation field), or else$...
substitutions would still need to use dynamic name lookups at rendering time. Whether the$
was required or optional in the${...}
form would be up to the i18n templating support functions.i18n"The result of adding ${x} to ${y} is ${x+y:expr_result}"
or_(t"The result of adding ${x} to ${y} is ${x+y:expr_result}")
would map to the English translation catalog entry"The result of adding $x to $y is $expr_result
". A regular specifier string could still be allowed after a second:
, since colons are permitted in specifier strings)You'd presumably get a minor performance win by replacing dynamic variable name lookups with compiler supported field interpolations, but even that is questionable since many (most?) i18n templates are interpolating local variable values that can be retrieved with a single dict lookup.
Instead, to get i18n use cases away from using dynamic string lookups, we'd likely need to define a dedicated "$-string" (dollar string) syntax that used PEP 292 interpolation syntax to define a
TemplateLiteral
instance. Such a syntax could also be really interesting for shell command execution.When discussing support for building lazy template field evaluation on top of the f-string inspired eager field evaluation, consider the following points:
format
builtin, and hence the default template renderer, supports()
as a format specifier on a field definition to indicate that the result should be called when rendering (allowing for convenient lazy evaluation with either alambda:
prefix or passing in a reference to an existing zero-argument callable).str.format
andstr.format_map
) rather than producing a fully resolved object in the initial rendering operation{-> expr}
is equivalent to{(lambda: expr)}
(syntax idea inspired by the syntax for return type annotations)Give examples, such as delaying expensive function calls when logging:
and naming fields in reusable SQL statements:
(SQL is an interesting case, since
executemany
specifically wants to give the DB API control of repeated substitutions so it can optimise things. Parameter substitution isn't just about avoiding SQL injections)(cc @nhumrich )