search

KittyCAD / modeling-app

The KittyCAD modeling app.
https://kittycad.io/modeling-app/download
MIT License
428 stars 37 forks source link

Add ability to iterate on design via Text-to-CAD API #4536

Open franknoirot opened 1 week ago

franknoirot commented 1 week ago

Add support for Text-to-CAD-supported design iterations within the modeling app.

Prerequisites

  1. This new API endpoint
  2. UI designs (TODO)

User flow

  1. User must have existing KCL code for this command to be available
  2. User can select an "Iterate with Text-to-CAD" command from the command palette[^1]
  3. User is taken through a flow as follows
    1. A selection step where any geometry that maps back to a code range[^2] can be selected.
      1. This step is optional. If nothing is selected, the whole file is operated on[^3]
    2. A prompt step with a text area like our current "Text-to-CAD" command[^4]
  4. This operation can take a while, so the user receives a loading toast until it fails or succeeds
  5. The operation's outcome is then show to the user[^5], where they can approve or reject it
    1. If it is approved, the new KCL code is applied[^6]

Notes

[^1]: We don't have toolbar buttons for generative AI functionality. Should we? [^2]: I don't know how this API would support default planes being selected, or other globally-available geometry [^3]: It is unclear to us what it means to pass along a selection. Does it limit the edits to that range, or just act as "areas of interest"? @Irev-Dev to talk to @greg-kcio about this. [^4]: We should consider renaming our current command to something more specific like "Generate with Text-to-CAD" [^5]: I believe this endpoint passes back a model that we can use the existing Text-to-CAD components to display [^6]: Do we attempt to preserve selections, or clear them after this application? Preserving them seems nontrivial

greg-kcio commented 1 week ago

Addressing the footnotes:

  1. We don't have toolbar buttons for generative AI functionality. Should we?

I want a toolbar icon (✨ anyone??) because it's a constant reminder that our AI features exist

  1. I don't know how this API would support default planes being selected, or other globally-available geometry

I think this won't be a problem since we can't select the default planes outside of the Start Sketch state. Are there other selectable features aside from the 6 default planes that are selectable but not represented in KCL?

  1. It is unclear to us what it means to pass along a selection. Does it limit the edits to that range, or just act as "areas of interest"? @Irev-Dev to talk to @greg-kcio about this.

The source range selection is context for the request, not a limit. For example:

Image

The KCL response for the prompt "add a button to this face" for this selection could add a buttonRadius parameter near the top of the code, outside the selection range.

Image

  1. We should consider renaming our current command to something more specific like "Generate with Text-to-CAD"

I dig it. For the new feature I like "Edit with Text-to-CAD" or "Iterate with Text-to-CAD"

  1. I believe this endpoint passes back a model that we can use the existing Text-to-CAD components to display

Right now the endpoint does not return an exported model, but we can change that if you want it.

  1. Do we attempt to preserve selections, or clear them after this application? Preserving them seems nontrivial

I'm on the fence. The user might want to make multiple consecutive edits to the same selection of features. But it might be possible for the edit to remove a selection.