The most common class of issues we see when using the pulumi-ai tools relates to the tool getting things wrong for less commonly used APIs or APIs that were introduced after the model training cutoff.
We expect to improve these cases by injecting pieces of Pulumi Schema into the prompt to help the AI to discover how to use them in-context.we can’t inject all schema for all resources though, so we expect to use a vector DB to lookup “relevant” schema and inject the top N of those in, so that the model can use that additional context as input to more accurately use the relevant APIs.
This also ensures that we can stay up-to-date with new API changes/additions.
We may in the future take this further and fine-tune a model on the additional schema data.
lukehoban
commented
1 year ago
The most common class of issues we see when using the
pulumi-aitools relates to the tool getting things wrong for less commonly used APIs or APIs that were introduced after the model training cutoff.We expect to improve these cases by injecting pieces of Pulumi Schema into the prompt to help the AI to discover how to use them in-context.we can’t inject all schema for all resources though, so we expect to use a vector DB to lookup “relevant” schema and inject the top N of those in, so that the model can use that additional context as input to more accurately use the relevant APIs.
This also ensures that we can stay up-to-date with new API changes/additions.
We may in the future take this further and fine-tune a model on the additional schema data.