scrapy-plugins / scrapy-playwright

šŸŽ­ Playwright integration for Scrapy
BSD 3-Clause "New" or "Revised" License
996 stars 110 forks source link

Task exception was never retrieved error #3

Closed jessrosenfield closed 3 years ago

jessrosenfield commented 3 years ago

The scraper's ability to output items seems unaffected, but at the end of the playwright request's response callback I get the following.

2021-01-26 11:58:58 machinename asyncio[21762] ERROR Task exception was never retrieved
future: <Task finished coro=<Route.continue_() done, defined at /Users/jessica/.local/share/virtualenvs/upwork-scrapers-2AkzeUYG/lib/python3.7/site-packages/playwright/async_api.py:481> exception=Error('Target page, context or browser has been closed')>
Traceback (most recent call last):
  File "/Users/jessica/.local/share/virtualenvs/upwork-scrapers-2AkzeUYG/lib/python3.7/site-packages/playwright/async_api.py", line 508, in continue_
    postData=postData,
  File "/Users/jessica/.local/share/virtualenvs/upwork-scrapers-2AkzeUYG/lib/python3.7/site-packages/playwright/_network.py", line 195, in continue_
    await self._channel.send("continue", cast(Any, overrides))
  File "/Users/jessica/.local/share/virtualenvs/upwork-scrapers-2AkzeUYG/lib/python3.7/site-packages/playwright/_connection.py", line 36, in send
    return await self.inner_send(method, params, False)
  File "/Users/jessica/.local/share/virtualenvs/upwork-scrapers-2AkzeUYG/lib/python3.7/site-packages/playwright/_connection.py", line 47, in inner_send
    result = await callback.future
playwright._types.Error: Target page, context or browser has been closed
althayr commented 3 years ago

Hi @jessrosenfield, i saw similar bugs when using the library.

One issue that i noticed is that the library is propagating Google Chrome errors, for example when visiting http://httpbin.org/status/404 scrapy throws a exception.

Screenshot from 2021-01-26 18-51-39

If the page you visit has other requests pending response when something similar happens, it can be the case that this propagates errors on other places (asyncio, playwright)

elacuesta commented 3 years ago

Hi, could you add a Minimal, Reproducible Example?

jessrosenfield commented 3 years ago

Whoops I'm the worst kind of user!! I unfortunately ended up scrapping my use of scrapy-playwright for that scraper since playwright ended up not being necessary.

I'll be using the package again and will provide necessary info if I come across a similar issue šŸ˜… in the meantime, at least other users know they're not the only one if they come across this. @althayr's suggestion that I may be closing the page with incomplete pending events seems like a plausible root cause.

Gallaecio commented 3 years ago

Closing due to lack of feedback.