search

reidbarber / gen-ui

Use text or image prompts to generate components and apps built with React.
11 stars 3 forks source link

Send screenshots after each update via GPT 4 Vision #14

Open Joabutt opened 9 months ago

Joabutt commented 9 months ago

Maybe you could use gpt 4 vision to check the current state of how the website looks, then send that input off together with the prompt so that the ai can understand what its looking at and what the user is referring to

reidbarber commented 9 months ago

I think this is a great idea. I see two areas this could fit in:

  1. It could do a direct follow up after each code modification to see if it did a good job matching the design request. It could then dispatch more actions to fix anything it missed.
  2. It would be good to supplement each generation so the model has a good understanding of the current state, without having to send all the code during each generation.

I think these would both rely on https://github.com/reidbarber/gen-ui/issues/9

Additionally, I think this would get much better results with when Assistants API adds support for Vision.