ulixee / hero

The web browser built for scraping
MIT License
649 stars 32 forks source link

How can I build a script using Hero into an exe without running into any issues? #180

Closed allendolgonos closed 1 year ago

allendolgonos commented 1 year ago

Tried to use pkg to build one of my Hero scripts into an exe. Got met with the following error code after launching the built exe:

Error: Cannot find module 'C:\snapshot\Allen\node_modules\@ulixee\hero-playground\index.cjs' 1) If you want to compile the package/file into executable, please pay attention to compilation warnings and specify a literal in 'require' call. 2) If you don't want to compile the package/file into executable and want to 'require' it from filesystem (likely plugin), specify an absolute path in 'require' call using process.cwd() or process.execPath. ←[90m at createEsmNotFoundErr (node:internal/modules/cjs/loader:960:15)←[39m ←[90m at finalizeEsmResolution (node:internal/modules/cjs/loader:953:15)←[39m ←[90m at resolveExports (node:internal/modules/cjs/loader:482:14)←[39m ←[90m at Function.Module._findPath (node:internal/modules/cjs/loader:522:31)←[39m ←[90m at Function.Module._resolveFilename (node:internal/modules/cjs/loader:919:27)←[39m at Function._resolveFilename (pkg/prelude/bootstrap.js:1819:46) ←[90m at Function.Module._load (node:internal/modules/cjs/loader:778:27)←[39m ←[90m at Module.require (node:internal/modules/cjs/loader:1005:19)←[39m at Module.require (pkg/prelude/bootstrap.js:1719:31) ←[90m at require (node:internal/modules/cjs/helpers:102:18)←[39m { code: ←[32m'MODULE_NOT_FOUND'←[39m, path: ←[32m'C:\snapshot\Allen\node_modules\@ulixee\hero-playground\package.json'←[39m, pkg: ←[33mtrue←[39m }

allendolgonos commented 1 year ago

I think I'm getting closer. Now I configured my package.json so that pkg is including everything in /node_modules. Now I am getting this error, browser opens but then closes after a few seconds.

2022-11-17T13:34:55.636Z ERROR [unblocked-agent-mitm-socket\lib\CertificateGenerator] CertsIpcHandler.onError { context: {}, sessionId: undefined, sessionName: undefined } Error: spawn C:\snapshot\Allen\Desktop\heroTest\node_modules\@ulixee\unblocked-agent-mitm-socket\dist\connect.exe ENOENT at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19) at onErrorNT (node:internal/child_process:477:16) at processTicksAndRejections (node:internal/process/task_queues:83:21) { errno: -4058, code: 'ENOENT', syscall: 'spawn C:\snapshot\Allen\Desktop\heroTest\node_modules\@ulixee\unblocked-agent-mitm-socket\dist\connect.exe', path: 'C:\snapshot\Allen\Desktop\heroTest\node_modules\@ulixee\unblocked-agent-mitm-socket\dist\connect.exe', spawnargs: [ '{"storageDir":"C:\\Users\\Allen\\AppData\\Local\\Temp/.ulixee","mode":"certs","debug":false,"ipcSocketPath":"\\\\.\\pipe\\ipc-certs-uRxvtaA3plQK8vrRLppl6"}' ] } Started Ulixee Miner at localhost:47875 2022-11-17T13:34:56.068Z ERROR [unblocked-agent-mitm-socket\lib\MitmSocketSession] ProxyIpcHandler.onError { context: { sessionId: 'nnChk7myYzcUBQao242g7' }, sessionId: 'nnChk7myYzcUBQao242g7', sessionName: 'default-session' } Error: spawn C:\snapshot\Allen\Desktop\heroTest\node_modules\@ulixee\unblocked-agent-mitm-socket\dist\connect.exe ENOENT at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19) at onErrorNT (node:internal/child_process:477:16) at processTicksAndRejections (node:internal/process/task_queues:83:21) { errno: -4058, code: 'ENOENT', syscall: 'spawn C:\snapshot\Allen\Desktop\heroTest\node_modules\@ulixee\unblocked-agent-mitm-socket\dist\connect.exe', path: 'C:\snapshot\Allen\Desktop\heroTest\node_modules\@ulixee\unblocked-agent-mitm-socket\dist\connect.exe', spawnargs: [ '{"rejectUnauthorized":false,"clientHelloId":"chrome-105","debug":false,"mode":"proxy","ipcSocketPath":"\\\\.\\pipe\\ipc-proxy-BJRnpx-KbxTDglNB8KwWf"}' ] } 2022-11-17T13:35:27.132Z ERROR [hero-core\index] UnhandledErrorOrRejection { context: {}, sessionId: null, sessionName: undefined } TimeoutError: Timeout waiting for "tab.goto(https://www.hibbett.com/)" at Timer.waitForPromise (C:\snapshot\Allen\Desktop\heroTest\node_modules\commons\lib\Timer.ts:55:21) at Page.goto (C:\snapshot\Allen\Desktop\heroTest\agent\core\lib\Page.ts:334:32) at Tab.goto (C:\snapshot\Allen\Desktop\heroTest\node_modules\core\lib\Tab.ts:453:28) at CommandRecorder.runCommandFn (C:\snapshot\Allen\Desktop\heroTest\node_modules\core\lib\CommandRecorder.ts:85:32) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async CommandRunner.runFn (C:\snapshot\Allen\Desktop\heroTest\node_modules\core\lib\CommandRunner.ts:36:14) at async ConnectionToHeroClient.executeCommand (C:\snapshot\Allen\Desktop\heroTest\node_modules\core\connections\ConnectionToHeroClient.ts:249:12) at async ConnectionToHeroClient.handleRequest (C:\snapshot\Allen\Desktop\heroTest\node_modules\core\connections\ConnectionToHeroClient.ts:64:14) ------REMOTE CORE--------------------------------- at Function.reviver (C:\snapshot\Allen\Desktop\heroTest\node_modules\commons\lib\TypeSerializer.ts:242:26) at JSON.parse () at Function.parse (C:\snapshot\Allen\Desktop\heroTest\node_modules\commons\lib\TypeSerializer.ts:31:17) at WsTransportToCore.onMessage (C:\snapshot\Allen\Desktop\heroTest\node_modules\net\lib\WsTransportToCore.ts:111:36) at WebSocket.emit (node:events:390:28) at Receiver.receiverOnMessage (C:\snapshot\Allen\Desktop\heroTest\node_modules\ws\lib\websocket.js:1068:20) at Receiver.emit (node:events:390:28) at Receiver.dataMessage (C:\snapshot\Allen\Desktop\heroTest\node_modules\ws\lib\receiver.js:517:14) at C:\snapshot\Allen\Desktop\heroTest\node_modules\ws\lib\receiver.js:468:23 at C:\snapshot\Allen\Desktop\heroTest\node_modules\ws\lib\permessage-deflate.js:308:9 ------CONNECTION---------------------------------- at new Resolvable (C:\snapshot\Allen\Desktop\heroTest\node_modules\commons\lib\Resolvable.ts:16:18) at createPromise (C:\snapshot\Allen\Desktop\heroTest\node_modules\commons\lib\utils.ts:168:10) at PendingMessages.create (C:\snapshot\Allen\Desktop\heroTest\node_modules\net\lib\PendingMessages.ts:47:44) at ConnectionToHeroCore.sendRequest (C:\snapshot\Allen\Desktop\heroTest\node_modules\net\lib\ConnectionToCore.ts:153:50) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async CoreCommandQueue.sendRequest (C:\snapshot\Allen\Desktop\heroTest\node_modules\client\lib\CoreCommandQueue.ts:287:12) at async Object.cb (C:\snapshot\Allen\Desktop\heroTest\node_modules\client\lib\CoreCommandQueue.ts:229:16) at async Queue.next (C:\snapshot\Allen\Desktop\heroTest\node_modules\commons\lib\Queue.ts:95:19) ------CORE COMMANDS------------------------------- at Queue.run (C:\snapshot\Allen\Desktop\heroTest\node_modules\commons\lib\Queue.ts:40:19) at CoreCommandQueue.run (C:\snapshot\Allen\Desktop\heroTest\node_modules\client\lib\CoreCommandQueue.ts:218:8) at CoreTab.goto (C:\snapshot\Allen\Desktop\heroTest\node_modules\client\lib\CoreTab.ts:243:36) at Tab.goto (C:\snapshot\Allen\Desktop\heroTest\node_modules\client\lib\Tab.ts:190:36) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async C:\snapshot\Allen\Desktop\heroTest\heroTest.js:16:5


------default-session----------------------------- ------nnChk7myYzcUBQao242g7-----------------------

blakebyrnes commented 1 year ago

Sorry for lack of reply here. I don't know how to help you out. Hero relies on several native executables that you'll need to figure out how to package for each respective operating system (unblocked/agent/mitm-socket/connect, Chrome (which currently defaults to installing into /ulixee/chrome/10X.X.X.X, and better-sqlite).

I think you might have better success copying what we are doing to distribute the ulixee/apps/desktop project. It's using electron-builder to compile dependencies for each OS.

odysseymemoirs commented 1 year ago

my suggestion is to consider all hero project as an pkg asset, the same as pkg works with native addons during packaging into executable and since hero depends on some go language executables you can not just launch exe app inside exe app because It needs to have a file on disk to load it

https://github.com/vercel/pkg/issues/1696 https://github.com/vercel/pkg/issues/141

blakebyrnes commented 1 year ago

Is there anything else to do on this ticket? I think you hopefully understand the pieces that need to be included in your build at least. I'm going to close since I don't know how to give any more help on this one. Feel free to ask questions on discord!