ulixee / hero

The web browser built for scraping
MIT License
649 stars 32 forks source link

hero/connections/ConnectionFactory #230

Closed benhichem closed 1 year ago

benhichem commented 1 year ago

so i have been using hero for the past year or so and lately when i install the @ulixee/hero & @ulixee/server package and lunch it like i usually and how its written in the documentation do am getting this error if anyone can help me solve this one i would be great full

@blakebyrnes @calebjclark @GlenDC @daleevans @jmannanc the error :

`2023-05-13T14:40:13.509Z ERROR [hero/connections/ConnectionFactory] Error connecting to core { context: {}, sessionId: null, sessionName: undefined } CanceledPromiseError: Websocket was not open at wsSend (/workspaces/CF-bigspy/node_modules/net/lib/WsUtils.ts:15:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async WsTransportToCore.send (/workspaces/CF-bigspy/node_modules/net/lib/WsTransportToCore.ts:52:7) at async Promise.all (index 1) at async ConnectionToHeroCore.sendRequest (/workspaces/CF-bigspy/node_modules/net/lib/ConnectionToCore.ts:158:24) at async ConnectionToHeroCore.afterConnect (/workspaces/CF-bigspy/node_modules/client/connections/ConnectionToHeroCore.ts:87:27) at async ConnectionToHeroCore.connect (/workspaces/CF-bigspy/node_modules/net/lib/ConnectionToCore.ts:83:11) 2023-05-13T14:40:13.514Z ERROR [hero-core/index] UnhandledError(fatal) { context: {}, sessionId: null, sessionName: undefined } CanceledPromiseError: Websocket was not open at wsSend (/workspaces/CF-bigspy/node_modules/net/lib/WsUtils.ts:15:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async WsTransportToCore.send (/workspaces/CF-bigspy/node_modules/net/lib/WsTransportToCore.ts:52:7) at async Promise.all (index 1) at async ConnectionToHeroCore.sendRequest (/workspaces/CF-bigspy/node_modules/net/lib/ConnectionToCore.ts:158:24) at async ConnectionToHeroCore.afterConnect (/workspaces/CF-bigspy/node_modules/client/connections/ConnectionToHeroCore.ts:87:27) at async ConnectionToHeroCore.connect (/workspaces/CF-bigspy/node_modules/net/lib/ConnectionToCore.ts:83:11) ------CORE COMMANDS------------------------------- at Queue.run (/workspaces/CF-bigspy/node_modules/commons/lib/Queue.ts:40:19) at CoreCommandQueue.run (/workspaces/CF-bigspy/node_modules/client/lib/CoreCommandQueue.ts:220:8) at Object.cb (/workspaces/CF-bigspy/node_modules/client/lib/CoreSessions.ts:35:64) at Queue.next (/workspaces/CF-bigspy/node_modules/commons/lib/Queue.ts:98:44) at Queue.run (/workspaces/CF-bigspy/node_modules/commons/lib/Queue.ts:43:10) at CoreSessions.create (/workspaces/CF-bigspy/node_modules/client/lib/CoreSessions.ts:34:8) at ConnectionToHeroCore.createSession (/workspaces/CF-bigspy/node_modules/client/connections/ConnectionToHeroCore.ts:67:38) at Hero._Hero_getCoreSessionOrReject (/workspaces/CF-bigspy/node_modules/client/lib/Hero.ts:686:10) at Hero.get activeTab [as activeTab] (/workspaces/CF-bigspy/node_modules/client/lib/Hero.ts:174:33) at Hero.goto (/workspaces/CF-bigspy/node_modules/client/lib/Hero.ts:465:17) ------AGENT QUEUE--------------------------------- at Queue.run (/workspaces/CF-bigspy/node_modules/commons/lib/Queue.ts:40:19) at CoreSessions.create (/workspaces/CF-bigspy/node_modules/client/lib/CoreSessions.ts:34:8) at ConnectionToHeroCore.createSession (/workspaces/CF-bigspy/node_modules/client/connections/ConnectionToHeroCore.ts:67:38) at Hero._Hero_getCoreSessionOrReject (/workspaces/CF-bigspy/node_modules/client/lib/Hero.ts:686:10) at Hero.get activeTab [as activeTab] (/workspaces/CF-bigspy/node_modules/client/lib/Hero.ts:174:33) at Hero.goto (/workspaces/CF-bigspy/node_modules/client/lib/Hero.ts:465:17) at X.$extract (file:///workspaces/CF-bigspy/src/index.js:65:24) at X.exec (file:///workspaces/CF-bigspy/src/index.js:54:16) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async file:///workspaces/CF-bigspy/src/index.js:72:1

node:internal/process/esm_loader:97 internalBinding('errors').triggerUncaughtException(`

blakebyrnes commented 1 year ago

The @ulixee/server package got renamed to @ulixee/cloud, so you're likely running on a version that's a few versions old. I think you're hitting a bug that we used to have here the cache directory/ulixee/hosts.json file wasn't getting cleaned up properly. You could try just deleting that file, or updating

benhichem commented 1 year ago

well the same thing ! after installing @ulixee/cloud am getting this error now ...

` Default Datastore Plugin not installed @ulixee/datastore-plugins-puppeteer-core/register Cannot find module '@ulixee/datastore-plugins-puppeteer-core/register' Require stack:

`

blakebyrnes commented 1 year ago

That warning message shouldn't be there... got added during last round of development. But the actual error there looks like a module didn't get installed correctly. Did you have errors during installation?

benhichem commented 1 year ago

i had few warning showing during installation yes !! " 4 critical severity vulnerabilities "

and after running npm audit

vm2 <3.9.17 Severity: critical vm2 Sandbox Escape vulnerability - https://github.com/advisories/GHSA-ch3r-j5x3-6q2m fix available vianpm audit fix --force` Will install undefined@undefined, which is a breaking change node_modules/@ulixee/datastore-core/node_modules/vm2 @ulixee/datastore-core Depends on vulnerable versions of vm2 node_modules/@ulixee/datastore-core @ulixee/cloud Depends on vulnerable versions of @ulixee/datastore-core Depends on vulnerable versions of @ulixee/datastore-plugins-hero-core node_modules/@ulixee/cloud @ulixee/datastore-plugins-hero-core * Depends on vulnerable versions of @ulixee/datastore-core node_modules/@ulixee/datastore-plugins-hero-core

4 critical severity vulnerabilities

To address issues that do not require attention, run: npm audit fix

To address all issues (including breaking changes), run: npm audit fix --force `

blakebyrnes commented 1 year ago

Hmm. I have a pending commit to that repo that the security audit fix seems to have unlinked from our vm2.. I tried re-synching their repo. Maybe that will fix it if you try again.

benhichem commented 1 year ago

i have deleted and reinstalled everything again and now there is a new error showing image

benhichem commented 1 year ago

i have deleted and reinstalled everything again and now there is a new error showing image

blakebyrnes commented 1 year ago

Sorry this is so nasty with that console.warn!! Will remove that in next version. This appears to actually just be an issue with your Chrome being unable to launch, so sorry for the hassle of updating Hero, but thanks for helping identity those 2 issues. If you're on linux, occasionally you need to install Chrome shared-library dependencies to get it working (or they change out underneath). Are there any detailed logs indicating why Chrome closed?

benhichem commented 1 year ago

this is the rest of the error, i will update my chrome and then see how is up !
image

blakebyrnes commented 1 year ago

The message should be on the CloudNode side

benhichem commented 1 year ago

where can i find them ?

benhichem commented 1 year ago

where can i find them ?

blakebyrnes commented 1 year ago

I just realized you're starting Cloud and Hero in the same process. You might see something by starting with DEBUG=ulx* as an env var

blakebyrnes commented 1 year ago

There's a dependency installer for your Chrome version at ~/.cache/ulixee/chromes//install-dependencies.deb

benhichem commented 1 year ago

well i have installed the Chrome versions from the path you provided and DEBUG=ulx*

its the same error tho : `

Default Datastore Plugin not installed @ulixee/datastore-plugins-puppeteer-core/register Cannot find module '@ulixee/datastore-plugins-puppeteer-core/register' Require stack:

blakebyrnes commented 1 year ago

Can you see if you can open the Chrome at that path (/home/snow/.cache/ulixee/chrome/112.0.5615.121/chrome)

/home/snow/.cache/ulixee/chrome/112.0.5615.121/chrome --headless --dump-dom https://ulixee.org

benhichem commented 1 year ago

no the chrome did no open !

benhichem commented 1 year ago

i have fixed that issues by reinstalling chrome by running npm i @ulixee/chrome-112-0 and now am able to start Hero ! but now am getting this if you do not mind expla image

blakebyrnes commented 1 year ago

I think that one is likely related to some code. Can you share the snippet failing or a session database? I'm on discord if you can't share the code for the part failing

benhichem commented 1 year ago

sure mate what is your discord ?