PromptTemplates should be template literals

[BLamy]

This library is cool but I feel like prompt templates should just use the language instead of being their own object.

https://www.typescriptlang.org/docs/handbook/2/template-literal-types.html

https://www.typescriptlang.org/play?#code/C4TwDgpgBACg9hAtgFTmAlgYwM5QLxQBEANnAG4SFQA+RAZgE7oQB2AJtgBbphW2F1SAdwgNshAFBSJoSLASIYDOIjDBkSMMQCGwaAQAGAdSZ6o2qGAXmARnACuwKABIA3vCSoMOAL4GpQA

Maybe use zod to enforce at runtime.

[cfortuner]

Hey! Thanks for the feedback.

What would the usage look like for this?

Definitely looking to simplify prompt creation for people so I like this direction.

[BLamy]

Right now when I call prompt.format I will get back text which is typed as a string then you pass that string to openAI's generate function.

I think it would be better if you wrapped .generate and made it generic. This will give you autocomplete of your prompts and a type level error if you try to generate a prompt that isn't allowed.

as const after the prompts is really the secret sauce here.

https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgAgDKoLLIN4ChkHIDmEI0ckAXMgBQAOUA9gLZ1jUDOYUoRAlMgC8APmRceIIrgC+uXAgA2cDh2QAFJqzBwARgogAeDSzYAVCKyWRVEAB6QQAE1XjeovIWQNgANwooFBWZqdAwAbjlPBEYQcQBXBDBGKBpGOlI4YBDMAQ9PQjAAC2AOADpA5iFkNIzgCM9ZfEISMih-ek02amMtc0t-Dlym-OQoCDA4qBBkIpLyoNKW8kgOkzA+esJZRtwAel31Rgt1TrBcMABPdMOLHrYq7Bm04ARqACIFRh8IN+QAH2QbxgPFIzmKdF+AKBnwA7tAOL9ZNFYmBkAgxv41EdmHcwH06FYUIJaI8knQXshpN1sbiBCJkAADADqPEgyDgXmx7J0jDiqIAJNgyS9pAz2apkVxzlcUFjbqd8YSqgAlcaTECmGUGS7pRgwNEYyBynEKiwE-zCPYHABSjAA1rLTtLrraHbiHsgAFb2iCa9LvGBxEAgC6QwGOOCOMNvRxxZg6RHyGJcA0QfyuiC4xX+Ko0R7eh1+lBU5AZ2lCUQM8yBZDMFAcwUF30y6Sln1i5Ro5NnHUoMum-ps4mqiZTIvamV61Ppn1Zs2Ey1ySWohhrXT6KpkGEnNd6QzGueDlAA-trbOQYQ0LfIADy6RAAEEAJI0ODkgDSEAufA2uFXWnXCBFlIZYIBodE00gU9enndp8x9It3kRH85FQoA

[cfortuner]

@BLamy

I really like the idea of constraining our inputs to only valid types, although it might be overkill at this stage.

I do think we should ensure that the generate function enforces the variables of a prompt are provided as a parameter.

We do this with generics today in the Prompt class, but I'm open to simplifying the Prompt class into functions like you proposed that take in variables instead.

Gonna leave this open for a bit and see if anyone else has input here.

[BLamy]

Sounds great. I'll keep playing around with things. My opinion may change once I have more context.

[emfastic]

@BLamy Agree with the idea of constraining inputs to only be valid types; right now I have limited utility in having typed variables like the PoemTopics example as most of what I'm doing is based on user inputs fed into a predefined prompt.

[BLamy]

@emfastic I'd argue you should be sanitizing your user input before feeding it into a predefined prompt. This approach really shines when you use zod to validate user input.

https://stackblitz.com/edit/node-ycgnwj?file=app.ts

[cfortuner]

The issue with this @BLamy , is that LLMs like openai are built to allow the user to write any string / text input and return valid text output.

For many use cases, constraining the inputs and output to specific text values doesn't make sense.

Instead, it would be more useful for validating that a Prompt is provided with the correct input variables based on the {{variable}} inside of the text.

Have a look at this: https://github.com/microsoft/prompt-engine

I'd love to hear if you have any opinions about it!

[BLamy]

The issue with this @BLamy , is that LLMs like openai are built to allow the user to write any string / text input and return valid text output.

This is taken from the midjourney docs. Both URLs and parameters could probably be validated using typescript and then leave the text prompt typed as string.

Each LLM also has a token limit. So you can do things like encode max lengths by creating some kind of zod-tiktoken library.

Even variables like "chat_history" typically follow a pattern that can be typed like

type chat_history = `USER: ${string}
LLM: ${string}`[]

Instead, it would be more useful for validating that a Prompt is provided with the correct input variables based on the {{variable}} inside of the text.

You can S.Split on {{ and then infer up until }}. TS Playground Example:

import { S } from 'ts-toolbelt'

type ExtractTemplateParams<TPath extends string> = {
  [K in S.Split<TPath, '{{'>[number] as K extends `${infer P}}}${string}`
    ? P
    : never]: string;
};

function format<T extends string>(
  prompt: T,
  params: ExtractTemplateParams<T>
) {}

const promptTemplate = 'Tell me a {{jokeType}} joke.';
format(promptTemplate, { jokeType: "test" })
format(promptTemplate, { test: 'asf' }); // <- type error
format(promptTemplate, {  }); // <- will have auto complete for key names

Have a look at this: https://github.com/microsoft/prompt-engine

Will take a look over the next few days

[cfortuner]

Okay now that is cool. Didn't know about Split but definitely interested in this now! Thanks for sharing this @BLamy

[BLamy]

I threw up a PR that uses the example above. https://github.com/cfortuner/promptable/pull/35

I've also found someone doing something similar with recursion. For Example

I'm also considering using HOTScript instead of ts-toolbox.

[yourbuddyconner]

Ok this is exactly what I have been ideating on how cool. I think leaning into the native type system to build a composable and type-safe prompt system is exactly why the choice of TS is a good one.

Already the simple prompt parameter name validation has been super-helpful, but I want to go 2-3 steps further in very much the same way as the original idea in this thread, though I am sure an abstraction can be achieved that uses less type arithmetic and is more intuitive to a JS programmer.

The pipeline I am working on presently is a GEtL (Generate, Extract/Transform, Load) pipeline and I am producing an interface at each step of generation such that the extracted output is constrained to the format the next step in the workflow expects such that the subsequent step can use output from the previous prompt in a highly structured way.

If I could pass the full complex input interface and a similar or identical output interface to the prompt engine that would be super sick and provide even more efficiency gains than the current prompt abstraction. From there it would be pretty trivial to generate prompts and types dynamically on the fly which opens up some real juicy ux opportunities.

[cfortuner]

Would love to see an example @yourbuddyconner

@BLamy we're working on adopting this 🙇‍♂️

[cfortuner]

We adopted the lastest feedback on GitHub! Closing this for now.