video to code generation shows empty in the UI, while the code was generated in the log. docker deployment to local macbook

[tianerphil]

Hi Abi, thank you for your work, it seems really interesting and well-rounded one. promising project. I ran it locally with docker installed on macbook pro. I can access the UI from localhost:5173, and it does work for static screenshot.

Nevertheless, when I use a short screen record video (8 MB, 1 min), it seems keep generating code during the full process around 3 minutes, it is proved by the backend log. but at the end of the processing, the UI suddenly cleared up, blanked with nothing showing up. there is no apparent error in the log. can you please share your opinion if you have encountered similar issue. thank you so much.

Local Deployment I followed the docker deployment instruction, only with these two customizations:

1. due to video to code feature involved. I have set api-key in the UI setting for anthropic api access. it seems it does work.
2. the default dockerfile for backend used python:3.12-slim as base layer, while I have changed it to python:3.10-slim, due to some incompatibility error happens when build the backend docker.

To Reproduce Steps to reproduce the behavior:

1. run the docker container locally with docker-compse up -d --build
2. visit the http://localhost:5173
3. give a short screen record video around 1 min, 8 MB
4. wait for around 3 min, there is nothing generated in the UI.

Screenshots of backend AND frontend terminal logs I have attached the screenshot of the final UI with empty generation result. I have attached backend and frontend log from the docker desktop for macOS.

backend-log-in-docker.txt frontend-log-in-docker.txt

[abi]

Thank you for the detailed description of your problem. Sorry for the slow response.

The first error I see in the backend logs is "websockets.exceptions.ConnectionClosedOK". This typically happens when the connection is lost between front-end and back-end for some reason.

The logs also show that a lot of code is being generated so video processing and prompt construction, etc. are all working correctly.

It's not immediately obvious to me what's wrong. But let's start with this: Try a really short video (like 5 seconds), and let me know if that works.

[tianerphil]

Hi Abi, sorry for the late response, I was on a business trip earlier. yes, it does work for a video of 5 sec, but since the video is short, the quality of the generated code might be compromised. but yes, there is no error this time. do you have any suggestion to prevent the long video error from happening again? thank you so much.

5sec-video-backend-log-in-docker-06-27.txt

[abi]

Sorry for the slow response - I've been traveling.

Hmm, that is strange. No matter the video length, the system generates 20 screenshots from the video. Re: quality of generations, video to code is still using Claude 3 Opus, not GPT4o and Claude 3.5 Sonnet which I'll switch to this week or next.

To help debug further, can you try your longer video again and share the logs again?

[tianerphil]

Dear Abi,

thank for the response, if the video processing basically is extracting 20 frames, does that mean, I can uplaod 20 static pictures to generate a dynamic react frontend code? if so, how to do it, thanks.

Best Regards, Liang Zhang 0509119456

Original

From:"Abi Raja"< @.*** >;

Date:2024/7/1 15:42

To:"abi/screenshot-to-code"< @.*** >;

CC:"Liang.Zhang"< @. >;"Author"< @. >;

Subject:Re: [abi/screenshot-to-code] video to code generation shows empty inthe UI, while the code was generated in the log. docker deployment to localmacbook (Issue #366)

Sorry for the slow response - I've been traveling.

Hmm, that is strange. No matter the video length, the system generates 20 screenshots from the video. Re: quality of generations, video to code is still using Claude 3 Opus, not GPT4o and Claude 3.5 Sonnet which I'll switch to this week or next.

To help debug further, can you try your longer video again and share the logs again?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[abi]

There isn't a way to upload multiple images at the moment. You'll have to use a video instead.