search

thp / urlwatch

Watch (parts of) webpages and get notified when something changes via e-mail, on your phone or via other means. Highly configurable.
https://thp.io/2008/urlwatch/
Other
2.81k stars 352 forks source link

Browser Job not working #728

Closed bokkabonga closed 1 year ago

bokkabonga commented 1 year ago

Hi Everyone,

i created my first browser job after isntalling pyppeeeer. Sadly running the job causes some kind of error. I couldn´t find any solution myself. So maybe someone here knows the problem and can make more sense of this error:/

`Exception while releasing resources for job: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 138, in test_filter raise job_state.exception File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 68, in enter self.job.main_thread_enter() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 377, in main_thread_enter self.ctx = BrowserContext() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 90, in init BrowserContext._BROWSER_LOOP = BrowserLoop() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 45, in init self._browser = self._event_loop.run_until_complete(self._launch_browser()) File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 51, in _launch_browser browser = yield from pyppeteer.launch() File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 307, in launch return await Launcher(options, **kwargs).launch() File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 168, in launch self.browserWSEndpoint = get_ws_endpoint(self.url) File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint raise BrowserError('Browser closed unexpectedly:\n') pyppeteer.errors.BrowserError: Browser closed unexpectedly:

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 78, in exit self.job.main_thread_exit() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 380, in main_thread_exit self.ctx.close() AttributeError: 'BrowserJob' object has no attribute 'ctx' Traceback (most recent call last): File "/usr/local/bin/urlwatch", line 11, in load_entry_point('urlwatch==2.24', 'console_scripts', 'urlwatch')() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/cli.py", line 112, in main File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 408, in run File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 210, in handle_actions File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 138, in test_filter File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 68, in enter File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 377, in main_thread_enter File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 90, in init File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 45, in init File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 51, in _launch_browser File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 307, in launch return await Launcher(options, **kwargs).launch() File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 168, in launch self.browserWSEndpoint = get_ws_endpoint(self.url) File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint raise BrowserError('Browser closed unexpectedly:\n') pyppeteer.errors.BrowserError: Browser closed unexpectedly:`

thp commented 1 year ago

Seems like this is a pyppeteer error:

pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Is there any other information printed somewhere?

Also, pyppeteer seems to be deprecated again, just to be replaced by playwright-python (we previously used a different library for browser jobs). Seems like maintaining a testing framework that hooks into a browser engine is something nobody wants to maintain?

In any case, can you check if running a simple Pyppeteer example works? (taken from the Pyppeteer README)

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://example.com')
    await page.screenshot({'path': 'example.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

If this doesn't work either, it might be due to your Pyppeteer installation, if this DOES work, we should investigate more what the issue could be.

bokkabonga commented 1 year ago

Sadly this didn´t work either:

Traceback (most recent call last): File "./test.py", line 11, in asyncio.get_event_loop().run_until_complete(main()) File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "./test.py", line 5, in main browser = await launch() File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 307, in launch return await Launcher(options, **kwargs).launch() File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 168, in launch self.browserWSEndpoint = get_ws_endpoint(self.url) File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint raise BrowserError('Browser closed unexpectedly:\n') pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Edit: Just noticed that pyppeteer was installed in Version 0.2.6 which is not the latest. I will see if I can update and if it changes anything

bokkabonga commented 1 year ago

Sadly 0.2.6 is the newest version i can install via pip. Installing via "pip install -U git+https://github.com/pyppeteer/pyppeteer@dev" sadly fails.

trevorshannon commented 1 year ago

I just did my first Browser job, and I got a similar error (Browser closed unexpectedly). Installing other dependencies and passing --no-sandbox to the pyppeteer launcher helped me resolve the issue. See:

Putting this here in case it helps anyone else!

bokkabonga commented 1 year ago

I just did my first Browser job, and I got a similar error (Browser closed unexpectedly). Installing other dependencies and passing --no-sandbox to the pyppeteer launcher helped me resolve the issue. See:

* https://stackoverflow.com/questions/61780476/pyppeteer-browser-closed-unexpectedly-in-aws-lambda/62305560#62305560

* https://stackoverflow.com/questions/57217924/pyppeteer-errors-browsererror-browser-closed-unexpectedly/71935536#71935536

Putting this here in case it helps anyone else!

The first link you provided fixed my error. In my case i had to install "libXScrnSaver.x86_64". Browser jobs now run without a problem. They don´t return what i wanted them to, but they work :D