MOST IMPORTANT: Add a repair flow

[ptmaroct]

Currently the tools work by splitting the project into js and html files Sometimes some components get broken and show up with some issues when webpage is viewed.

You can view all generated websites here: https://shipstation.ai/all

Check out as many websites as you can and spot whatever issues you find and a patttern of where AI is breaking or giving poor output.

Effectively, we need to add a "analyse and repair" website tool. One of the approach for the tool is:

1. Once website is generated, open the website in a headless browser programmatically and take screenshots
2. Send those screenshots for analysis to AI which detects potentially broken components like features/pricing/footer etc
3. Rewrite the code for the component if needed
4. Call project deployed event once this all is done

We need to aim to fix both visual aspect of things and try to fix broken carousels, buttons etc which are supposed to do work but dont. Text and other colors used should also have good contrast ration with the backgrounds as well. You

We are open to any other approach as well which is faster, smarter and more reliable.

[ptmaroct]

Example: https://shipstation.ai/site/bananacoin-landing-page-iKmTSSoX/

The header is broken for this website

[pothur9]

i have found few pages with broken components

https://shipstation.ai/site/yuvraj-singh-dashboard-portfolio-WNJ6quXd/ page not loading

https://shipstation.ai/site/n-times-y-curiosity-app-landing-page-5DR1kO8q/ header is broken

https://shipstation.ai/site/siddharth-gangal-portfolio-room_c9ysaewn0/ header is broken and page not loading

https://shipstation.ai/site/siddharth-gangal-portfolio-room_rnmcn4dg1/ header and main section broken

https://shipstation.ai/site/newslens-balanced-keyword-tracker-room_2g2m6dkmz/ full page is broken

https://shipstation.ai/site/fashion-expression-waitlist-landing-page-room_6wlb8y16e/ header and body broken

https://shipstation.ai/site/yuvraj-singh-dashboard-portfolio-room_rwyep4ewp/ cards size and alignment problem

https://shipstation.ai/site/the-solar-labs-landing-page-room_2a9ibtzlx/ full page is broken

https://shipstation.ai/site/sangeet-singh-s-hippie-frontend-portfolio-room_9cx1z2jxc/ page not loading

https://shipstation.ai/site/captivacrm-room_3z31xc3e2/ header and pricing section broken

[pothur9]

i have the above issues could you please explain how to fix this

[vikaswakde]

Here is the possible approach i write code for :

1. Capture screenshots after deployment (using a puppeteer)
2. Analyze the screenshots (using Google Vision API)
3. Repair the website if issues are detected (repair service Function)
4. Re-deploy the website after repairs
5. Capture new screenshots after repairs
6. Re-analyze the website and final deploy

[ujjwal-45]

That's a great approach @vikaswakde , but how will google vision identify the broken components.

[vikaswakde]

That's a great approach @vikaswakde , but how will google vision identify the broken components.

Google Cloud Vision API doesn't directly identify broken components or layout issues. Instead, it provides us with information about the contents of the image, which we then interpret to infer potential issues. Here's how it actually works: Object Detection: The Vision API identifies objects in the image. We use this to check for the presence of expected elements like headers, navigation bars, buttons, etc. Text Detection: It extracts any text present in the image. This helps us verify if expected content is present. Label Detection: It provides general labels describing the image content.

 async function detectIssues(analysisResults) {
  const issues = [];

  // Check for missing key elements
  const expectedElements = ["header", "navigation", "button", "text", "image"];
  expectedElements.forEach((element) => {
    if (!analysisResults.objects.includes(element.toLowerCase())) {
      issues.push(`Missing ${element}`);
    }
  });

  // Check for potential layout issues
  if (analysisResults.objects.length < 5) {
    issues.push("Potential layout issues - few elements detected");
  }

  // Check for text content
  if (!analysisResults.text) {
    issues.push("No text detected - potential content loading issue");
  }

  return issues;
} 

This function infers potential issues by: Checking if expected elements are present in the detected objects. Assuming potential layout issues if very few objects are detected. Flagging a potential content loading issue if no text is detected.

this is just one possible way, there might be more efficient ways, this is just what I got the code for, anyways keep looking for more ways

[ujjwal-45]

@vikaswakde got it.

[pramaths]

@vikaswakde are u working on this?

[vikaswakde]

@pramaths I have proposed one possible approach, If they like it, and it works in prod then it's good,

Otherwise, if you find, a more better and faster approach, you can raise the pr

[pramaths]

@vikaswakde why vision API from Google why not Claude vison API?

[vikaswakde]

hey @pramaths sorry for the late reply, after reading about Cluade Vision API, I think lt could indeed be a better choice for this project, I have never worked with vision API before, it's just this issue I searched for a solution and implemented it,

but yes Claude's vision could be better suited for this project not Google's vision

[pramaths]

@vikaswakde are u able to extract images from given website?

[vikaswakde]

@vikaswakde are u able to extract images from given website?

you mean screenshots ?

not yet

[pramaths]

@vikaswakde I am able extract the images using playwright/puppeteer from the URL, now the problem is the API, to generate a new component we need paid API, free API doesn't work (tokens are not enough), with free plan

[vikaswakde]

@vikaswakde I am able extract the images using playwright/puppeteer from the URL, now the problem is the API, to generate a new component we need paid API, free API doesn't work (tokens are not enough), w

https://github.com/daytimedrinkingclub/shipstation/pull/35#issuecomment-2293217668