ulixee / hero

The web browser built for scraping
MIT License
647 stars 32 forks source link

Execute js on page? #267

Closed JortsEnjoyer0 closed 3 weeks ago

JortsEnjoyer0 commented 3 weeks ago

Is there really no way to execute javascript on a page? If not, I assume because of detection reasons? Javascript should not affect detection right? Chrome extensions run javascript all the time on pages and they don't flag the browser as a bot, so why couldn't we do the same with Hero?

blakebyrnes commented 3 weeks ago

You can use the execute-js plugin to activate running javascript on page. And if you're only grabbing a variable, you can run the frame/tab.getJsValue(). It will use the same thing as extensions to run in an isolated "dom world" if you want it to so it does the same thing.

The main reasons we don't have it installed by default are that:

  1. All the tooling we built (like Replay, FlowEvents, etc) don't work when you jump into raw page level javascript
  2. It tends to make the code hard to follow with the context switching (eg, you create a variable and expect it to exist in your dom level code, but your code was actually serialized into a string and run in the browser)
JortsEnjoyer0 commented 3 weeks ago

Gotchya, I appreciate the explanation.