abi / screenshot-to-code

Drop in a screenshot and convert it to clean code (HTML/Tailwind/React/Vue)
https://screenshottocode.com
MIT License
64.32k stars 7.84k forks source link

Hallucination #448

Open lionelcoding opened 6 hours ago

lionelcoding commented 6 hours ago

Describe the bug When using Screenshot to Code, the generated code output is incorrect/hallucinates and does not accurately match the expected layout and components of the original screenshot. The AI seems to be generating a news website layout instead of a Google Meet interface.

To Reproduce Steps to reproduce the behavior:

Go to Screenshot to Code interface Upload a screenshot of Google Meet interface Select "HTML + Tailwind" as the generation option Click "Update" or "Regenerate" Observe that the generated result shows a news website layout with articles about "Israeli Military Raids" and "Las Vegas Weddings" instead of the Google Meet interface components Screenshots of backend AND frontend terminal logs From the provided screenshot, we can see:

Input: Google Meet interface screenshot showing a video call interface with control buttons and a participant Output: Incorrectly generated layout showing a news website with unrelated content The generation settings show "HTML + Tailwind" is selected No terminal logs are visible in the provided screenshot to help debug the issue Additional notes:

This appears to be a model hallucination issue where the AI is not correctly interpreting the input screenshot The generated layout is completely different from the source material (Google Meet interface) The issue seems to be with the model's interpretation rather than a technical error, as the generation process completes but produces incorrect results CleanShot 2024-12-02 at 00 57 12@2x

abi commented 4 hours ago

Do you have the SHOULD_MOCK_AI_RESPONSE env variable set in your backend .env file? The response here is a mock response that is used for testing during development.