Template inconsistency C# and Extension

microsoft / prompty

Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.

https://prompty.ai

MIT License

459 stars 36 forks source link

Template inconsistency C# and Extension #112

Open injeniero opened 2 days ago

injeniero commented 2 days ago

I have been using Prompty to write the prompts for our C# Unity game. A template that does work in VS does not work in c#. After some research, I noticed the problem is the Template engine used.

On VS the extension is using nunjucks while the C# runtime is using scriban with support for (liquid)[https://shopify.github.io/liquid/]. So the problem is all of those template engines share a basic template grammar, but behavior and operators are all over the place.

A few examples:

Nunjucks only has the filter length, while liquid only has size.
Nunjucks | join(", "), while liquid | join: ", "
Nunjucks has the not unary operator, while liquid does not.
Expressions precedence, I had to add parenthesis for C# to produce the same output. This causes the Editor to complain when trying to extract semantic tokens, but it does render the template correctly.

I have not used python runtime, but I checked it is using jinja2 which I'm sure has its own different behaviors.

injeniero commented 2 days ago

To move forward, I'm using a custom version of Scriban where I added support for the missing filters that I'm currently using and the not operator. I'm not using filters with parameters because I don't know if it is worth the effort to keep modifying Scriban to also support that.

sethjuarez commented 1 day ago

Yeah - we've been noodling over implementing an "uber" template language just for these things so all the languages can be unified. In general, you can override the template language and parser mechanism with custom invokers (here's the Liquid one in the new Core runtime we are working on). Would love your thoughts on how we might move forward on this!

injeniero commented 1 day ago

I was also looking for options and there is no template with logic that works across python, c# and JS.

Possible solutions I see are:

Make the extension capable of using an installed runtime. Most people would use VS Code to edit their prompts while having the proper runtime in place. So if I'm targeting C#, then I can select C# runtime to be used by the extension, same for python or web. I would make the template field required so you can check if the selected runtime supports that template.
Work on a template engine that can be run in different runtimes. This is a larger effort. In this case I would try to write the engine in C/C++ so it can be accessed from python, c#, Node with native bindings and also provide Wasm builds to be used in the Web.
Same as previous but using jinja2. There is a C++ port for Jinja2 port that is working and is actively in development. No idea how easy is to make it Wasm compatible.