search

pulumi / pulumi-ai

223 stars 16 forks source link

Asking for C# code generates C# but TS project #89

Open arturl opened 3 months ago

arturl commented 3 months ago

What happened?

  1. Navigate to https://www.pulumi.com/ai
  2. Without changing the language at the bottom dialog (i.e. leaving TypeScript) type "C# code for Azure web application"

The tool generates a C# program while still saying "TypeScript":

image

  1. Now click the "Deploy with Pulumi" button: image

  2. Go through the project creation steps. The project is generated as TypeScript with C# code:

image

Example

See above

Output of pulumi about

not relevant

Additional context

While this is a user error (I should have switched the language), the experience is a trap. I explicitly asked for what I want (a C# app) and expect to get it.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

AaronFriel commented 3 months ago

Curious what you would expect in this scenario, I think we have a pretty wide array of options:

  1. Pulumi AI would override your chosen language based on the prompt - and what would the boundaries of that be, e.g.: "Write a program to deploy a C# web app" would be ambiguous?
    1. Should we remove the language selection buttons entirely, and determine how to handle if the user asks for "a program" without a language in their prompt?
  2. Pulumi AI should refuse to generate a program for "the wrong" language, be prompted to detect this situation
arturl commented 3 months ago

The least surprising behavior would be to switch the language selector to the user-specified language. Internally, I would add a sanity check to detect language mismatch between the generated language and the file type. As in, if we're writing C# code into a .ts file, fail hard and report an error. (Not sure if this is recoverable, if we know the user intent at that point)