search

K-RT-Dev / VGT

Program to translate Japanese text through image recognition and GPT 3.5
MIT License
76 stars 6 forks source link

Can't capture screen shot #1

Open huyllc opened 1 year ago

huyllc commented 1 year ago

Hi, when I try to capture screenshot it show an error like this 

[1] Error occurred in handler for 'captureScreenshot': TypeError: Error processing argument at index 2, conversion failure from 
[1]     at node:electron/js2c/browser_init:177:945
[1]     at new Promise (<anonymous>)
[1]     at Object.t.getSourcesImpl (node:electron/js2c/browser_init:177:697)
[1]     at Object.getSources (node:electron/js2c/browser_init:45:277)
[1]     at C:\Users\LENOVO\Desktop\ImageDetect\TranslateManga\VGT-master\electron\handlers\ipcHandler.js:138:45
[1]     at node:electron/js2c/browser_init:201:579
[1]     at Object.<anonymous> (node:electron/js2c/browser_init:165:10272)
[1]     at Object.emit (node:events:394:28)

can you help me

K-RT-Dev commented 1 year ago

I'm working on trying to replicate your error

Few questions: Does the error occur when using the compiled version (.exe) or running the source code? Is the error consistent? Does it continue to happen even if you restart the software or try to take other screenshots?

rDarge commented 1 year ago

I've started getting this issue consistently since June 1st, when I tried to pull my fork down to a new environment. It's happened both on Linux and Windows, though it was working in both prior to some undetermined point after which I haven't been able to capture another screenshot. 

[1] Error occurred in handler for 'captureScreenshot': TypeError: Error processing argument at index 2, conversion failure from 
[1]     at node:electron/js2c/browser_init:177:945
[1]     at new Promise (<anonymous>)
[1]     at Object.t.getSourcesImpl (node:electron/js2c/browser_init:177:697)
[1]     at Object.getSources (node:electron/js2c/browser_init:45:277)
[1]     at C:\Users\Me\workspace\ORIGINAL-VGT\electron\handlers\ipcHandler.js:143:45
[1]     at node:electron/js2c/browser_init:201:579
[1]     at Object.<anonymous> (node:electron/js2c/browser_init:165:10272)
[1]     at Object.emit (node:events:394:28)

Here's a link to my package-lock.json: https://gist.github.com/rDarge/42b2a709d37bfd2b3052d72a1ce7237e

I was actually trying to boot up a version of VGT I had working so I could add screenshots for some of the issues I was wanted to make, but wasn't able to actually take a screenshot.

rDarge commented 1 year ago

It was a huge pain troubleshooting this, but I think I've actually figured it out. If you supply a number with a decimal point in one of the parameters for thumbnailSize in desktopCapturer.getSources, you will hit this error. I'm able to avoid it by applying a Math.floor operation to both screenWidth and screenHeight in ipcHandler.js in the captureScreenshot handler.

Here are some logs I pulled from testing this, and a fix I've put in place. 

(manga-ocr-fast-api-py3.9) PS C:\Users\Ryan\workspace\VGT> npm run electron-dev

> VGT@0.1 electron-dev
> cross-env NODE_ENV=development concurrently "webpack serve --config webpack.dev.config.js"  "electron ."

[1] 
[1] INFO:     Will watch for changes in these directories: ['C:\\Users\\Ryan\\workspace\\VGT']
[1]
[1] INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[1] INFO:     Started reloader process [11556] using WatchFiles
[1]
[0] <i> [webpack-dev-server] Project is running at:
[0] <i> [webpack-dev-server] Loopback: http://localhost:3000/
[0] <i> [webpack-dev-server] On Your Network (IPv4): http://192.168.1.26:3000/
[0] <i> [webpack-dev-server] Content not from webpack is served from 'C:\Users\Ryan\workspace\VGT\dist' directory
[1] INFO:     Started server process [11112]
[1] INFO:     Waiting for application startup.
[1]
[1] INFO:     Application startup complete.
[1]
[1] INFO:     127.0.0.1:58653 - "GET /check HTTP/1.1" 200 OK
[1]
[1] INFO:     127.0.0.1:58654 - "GET /modelCheck HTTP/1.1" 200 OK
[1]
[1] 2023-06-03 12:03:58.237 | INFO     | manga_ocr.ocr:__init__:13 - Loading OCR model from kha-white/manga-ocr-base
[1]
[1] C:\Users\Ryan\AppData\Local\pypoetry\Cache\virtualenvs\manga-ocr-fast-api-PMbG7k51-py3.9\lib\site-packages\transformers\models\vit\feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead.
[1]   warnings.warn(
[1]
[0] <i> [webpack-dev-middleware] wait until bundle finished: /index.html
[1] 2023-06-03 12:04:01.017 | INFO     | manga_ocr.ocr:__init__:22 - Using CPU
[1]
[1] 2023-06-03 12:04:01.751 | INFO     | manga_ocr.ocr:__init__:29 - OCR ready
[1]
[1] loadMangaOCR
[1] INFO:     127.0.0.1:58655 - "POST /loadMangaOCR HTTP/1.1" 200 OK
[1]
[0] asset bundle.js 9.13 MiB [emitted] (name: main)
[0] asset index.html 233 bytes [emitted]
[0] orphan modules 3.01 MiB [orphan] 2041 modules
[0] runtime modules 28.9 KiB 16 modules
[0] built modules 3.06 MiB [built]
[0]   modules by path ./node_modules/ 2.95 MiB 606 modules
[0]   modules by path ./react/ 114 KiB
[0]     modules by path ./react/components/*.js 110 KiB
[0]       ./react/components/App.js 17.1 KiB [built] [code generated]
[0]       ./react/components/Main.js 4.64 KiB [built] [code generated]
[0]       + 8 modules
[0]     modules by path ./react/*.css 3.65 KiB
[0]       ./react/App.css 2.23 KiB [built] [code generated]
[0]       ./node_modules/css-loader/dist/cjs.js!./react/App.css 1.42 KiB [built] [code generated]
[0]     ./react/index.js 330 bytes [built] [code generated]
[0]   external "events" 42 bytes [built] [code generated]
[0]   external "electron" 42 bytes [built] [code generated]
[0] webpack 5.76.2 compiled successfully in 6251 ms
[1] 2256 1504.5
[1] Error occurred in handler for 'captureScreenshot': TypeError: Error processing argument at index 2, conversion failure from
[1]     at node:electron/js2c/browser_init:177:945
[1]     at new Promise (<anonymous>)
[1]     at Object.t.getSourcesImpl (node:electron/js2c/browser_init:177:697)
[1]     at Object.getSources (node:electron/js2c/browser_init:45:277)
[1]     at C:\Users\Ryan\workspace\VGT\electron\handlers\ipcHandler.js:144:45
[1]     at node:electron/js2c/browser_init:201:579
[1]     at Object.<anonymous> (node:electron/js2c/browser_init:165:10272)
[1]     at Object.emit (node:events:394:28)
Terminate batch job (Y/N)? [0] <i> [webpack-dev-server] Gracefully shutting down. To force exit, press ^C again. Please wait...
[1] C:\Users\Ryan\workspace\VGT\node_modules\electron\dist\electron.exe exited with signal SIGINT
[1] electron . exited with code 1
[0] webpack serve --config webpack.dev.config.js exited with code 0

Note the following line, which is a console log of screenWidth and screenHeight, respectively:

[1] 2256 1504.5

In ipcHandler.js I made the following changes to lines 138-140:

      const screenWidth = Math.floor(screenDetails.size.width * screenDetails.scaleFactor);
      const screenHeight =
        Math.floor(screenDetails.size.height * screenDetails.scaleFactor)

After applying this change I was able to take a screenshot again. 

(manga-ocr-fast-api-py3.9) PS C:\Users\Ryan\workspace\VGT> npm run electron-dev

> VGT@0.1 electron-dev
> cross-env NODE_ENV=development concurrently "webpack serve --config webpack.dev.config.js"  "electron ."

[1] 
[1] INFO:     Will watch for changes in these directories: ['C:\\Users\\Ryan\\workspace\\VGT']
[1]
[1] INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[1] INFO:     Started reloader process [2148] using WatchFiles
[1]
[0] <i> [webpack-dev-server] Project is running at:
[0] <i> [webpack-dev-server] Loopback: http://localhost:3000/
[0] <i> [webpack-dev-server] On Your Network (IPv4): http://192.168.1.26:3000/
[0] <i> [webpack-dev-server] Content not from webpack is served from 'C:\Users\Ryan\workspace\VGT\dist' directory
[1] INFO:     Started server process [15656]
[1] INFO:     Waiting for application startup.
[1]
[1] INFO:     Application startup complete.
[1]
[1] INFO:     127.0.0.1:53409 - "GET /check HTTP/1.1" 200 OK
[1]
[1] INFO:     127.0.0.1:53410 - "GET /modelCheck HTTP/1.1" 200 OK
[1]
[1] 2023-06-03 12:05:07.255 | INFO     | manga_ocr.ocr:__init__:13 - Loading OCR model from kha-white/manga-ocr-base
[1]
[1] C:\Users\Ryan\AppData\Local\pypoetry\Cache\virtualenvs\manga-ocr-fast-api-PMbG7k51-py3.9\lib\site-packages\transformers\models\vit\feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead.
[1]   warnings.warn(
[1]
[0] <i> [webpack-dev-middleware] wait until bundle finished: /index.html
[1] 2023-06-03 12:05:10.143 | INFO     | manga_ocr.ocr:__init__:22 - Using CPU
[1]
[1] 2023-06-03 12:05:10.797 | INFO     | manga_ocr.ocr:__init__:29 - OCR ready
[1]
[1] loadMangaOCR
[1] INFO:     127.0.0.1:53411 - "POST /loadMangaOCR HTTP/1.1" 200 OK
[1]
[0] asset bundle.js 9.13 MiB [emitted] (name: main)
[0] asset index.html 233 bytes [emitted]
[0] orphan modules 3.01 MiB [orphan] 2041 modules
[0] runtime modules 28.9 KiB 16 modules
[0] built modules 3.06 MiB [built]
[0]   modules by path ./node_modules/ 2.95 MiB 606 modules
[0]   modules by path ./react/ 114 KiB
[0]     modules by path ./react/components/*.js 110 KiB
[0]       ./react/components/App.js 17.1 KiB [built] [code generated]
[0]       ./react/components/Main.js 4.64 KiB [built] [code generated]
[0]       + 8 modules
[0]     modules by path ./react/*.css 3.65 KiB
[0]       ./react/App.css 2.23 KiB [built] [code generated]
[0]       ./node_modules/css-loader/dist/cjs.js!./react/App.css 1.42 KiB [built] [code generated]
[0]     ./react/index.js 330 bytes [built] [code generated]
[0]   external "events" 42 bytes [built] [code generated]
[0]   external "electron" 42 bytes [built] [code generated]
[0] webpack 5.76.2 compiled successfully in 6233 ms
[1] 2256 1504
[1] Guardando nueva imagen
[1] Guardando texto detectado
[1] INFO:     127.0.0.1:64460 - "POST /translateDataUrlImg HTTP/1.1" 200 OK
[1]
[1] INFO:     127.0.0.1:64463 - "POST /translateText HTTP/1.1" 200 OK
[1]
[1] Guardando texto traducido

screenWidth and screenHeight are even numbers now.

[1] 2256 1504

I'll dig into this a bit more so I understand the implications of fractional screen width/height calculations, but as a stopgap fix flooring these numbers has worked fine for me, with no other issues.

K-RT-Dev commented 1 year ago

Good job finding that bug! Yes, I think that must be it. I never tested the system with operating systems that are using additional scaling.

I will check that everything continues to work correctly and I accept your PR.

I have never tested the program on Linux. So, as you mentioned, you were able to run it correctly, right? It would be great to make an installable version.

Thank you very much for your work

rDarge commented 1 year ago

No problem. With some tweaks I was able to run it on linux, and OSX as well, but I don't have immediate access to an OSX build environment currently. I'll open up another issue for linux compatibility and prepare a PR for that soon.