search

thp / urlwatch

Watch (parts of) webpages and get notified when something changes via e-mail, on your phone or via other means. Highly configurable.
https://thp.io/2008/urlwatch/
Other
2.79k stars 350 forks source link

Proposals for "navigate": ability to set a custom user agent and to execute a javascript in the page context #670

Open maiux opened 2 years ago

maiux commented 2 years ago

I like this project, if you agree I think this two features will be an improvement, since some websites knock down the request when they spot an "*/headless" user agent.

I hope it will be easy to set a custom user agent and evaluate javascript in the _get_content() function of browser module using page.setUserAgent and page.evaluate method exposed.

I think also it will be a great addition using playwright as an alternative browser interface, since playwright IMHO has a much smoother experience handling automation and has a native python API

thp commented 2 years ago

Both suggestions sound fine to me, patches gladly accepted.

fversaci commented 2 years ago

In #700 I added an optional key for changing the useragent (but did nothing about the javascript evaluation).