search

scrapy-plugins / scrapy-playwright

🎭 Playwright integration for Scrapy
BSD 3-Clause "New" or "Revised" License
954 stars 104 forks source link

ERROR: Task was destroyed but it is pending! #188

Open ma-pony opened 1 year ago

ma-pony commented 1 year ago

I sometimes get this error when i use scrapy-pilaywright

2023-03-31 09:33:35 [asyncio] ERROR: Task was destroyed but it is pending!
source_traceback: Object created at (most recent call last):
  File "/Users/rccpony/PycharmProjects/bid-spiders/packages/universal_spiders/main.py", line 5, in <module>
    cmdline.execute(
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 154, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 109, in _run_print_help
    func(*a, **kw)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 162, in _run_command
    cmd.run(args, opts)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/commands/crawl.py", line 27, in run
    self.crawler_process.start()
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/crawler.py", line 363, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/twisted/internet/asyncioreactor.py", line 257, in run
    self._asyncioEventloop.run_forever()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1901, in _run_once
    handle._run()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 249, in run
    await self._transport.run()
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_transport.py", line 155, in run
    self.on_message(obj)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 210, in <lambda>
    self._transport.on_message = lambda msg: self.dispatch(msg)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 356, in dispatch
    object._channel.emit(
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 176, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 154, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py", line 48, in _emit_run
    coro: Any = f(*args, **kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 131, in <lambda>
    lambda params: self._on_request(
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 485, in _on_request
    page.emit(Page.Events.Request, request)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 176, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 154, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py", line 57, in _emit_run
    fut: Any = ensure_future(cast(Any, coro), loop=self._loop)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 615, in ensure_future
    return _ensure_future(coro_or_future, loop=loop)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 636, in _ensure_future
    return loop.create_task(coro_or_future)
task: <Task pending name='Task-386' coro=<_make_request_logger.<locals>._log_request() running at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:429> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py:168] created at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py:636>
[scrapy.utils.log] INFO: Scrapy 2.7.0 started (bot: universal_spiders)
[scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.13, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 21.7.0, Python 3.10.10 (v3.10.10:aad5f6a891, Feb  7 2023, 08:47:40) [Clang 13.0.0 (clang-1300.0.29.30)], pyOpenSSL 22.1.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 38.0.4, Platform macOS-13.2.1-arm64-arm-64bi

(Edit) Reproducible example at #233

stefanondisponibile commented 1 year ago

Encountering the same issue here.

2023-05-24 09:10:35 [scrapy.core.engine] INFO: Spider closed (finished)
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing download handler
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing browser
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing download handler
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing browser
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-5112' coro=<_make_request_logger.<locals>._log_request() running at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/playwright/_impl/_connection.py:168]>
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-5113' coro=<_make_request_logger.<locals>._log_request() running at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/playwright/_impl/_connection.py:168]>
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
...[many more here]

Logs seem to point here (using scrapy-playwright 0.0.26) but when I read this question I started wondering if this could be related too.

A nice starting point would be understanding if this is a scrapy-playwright or playwright kind of issue. What do you think?

auxsvr commented 1 year ago

This seems to be related to https://github.com/jfhbrook/pyee/issues/120. Keeping a reference to the tasks prevents this in my tests.

elacuesta commented 1 year ago

Please share a minimal, reproducible example.

ma-pony commented 1 year ago

Please share a minimal, reproducible example.

This was an occasional issue, the code that caused the problem was removed by me at the time, and i will try to reproduce the problem recently

elacuesta commented 11 months ago

Using the example from #233 I'm seeing much less of these messages after upgrading pyee from 9.0.4 to 11.0.0.

mgrist commented 11 months ago

@elacuesta Any luck on a solution? I have a long-running scraper with this issue as well. It's not consistent and happens intermittently. Sometimes errors in 10 minutes, sometimes after 4 hours.

I just took your advice and upgraded my Pyee version to 11.0, I will try again.

elacuesta commented 11 months ago

@elacuesta Any luck on a solution?

I don't have plans to work on this, from what I see it's an upstream issue.

ThinksFast commented 9 months ago

I'm getting a ton of these errors all of a sudden. I was running scrapy + playwright without issue for several months. Was this issue raised with Pyee?

For reference, my environment is using: python 3.11.6 scrapy 2.11.0 scrapy-playwright 0.0.33 twisted 22.10.0 pyee 11.0.1

Example logs:

2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4511' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4514' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4512' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4515' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4763' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4766' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4764' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4767' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4772' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4773' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4797' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4798' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4809' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4812' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4810' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4813' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4863' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4864' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4869' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4872' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4875' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4870' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4873' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4878' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4881' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4879' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4882' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4884' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4887' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4885' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4888' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4876' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4931' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4932' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
mgrist commented 9 months ago

@ThinksFast I was never able to solve the issue here. I ended up migrating to scrapy-splash.

ThinksFast commented 9 months ago

@ThinksFast I was never able to solve the issue here. I ended up migrating to scrapy-splash.

Thanks for the follow up! I found out earlier today I was getting this issue due to an exception in my Scrapy pipeline file. The pipeline function to save the data was throwing an exception due to Unique Constraints set up on the table columns. But this exception wasn't getting logged for some reason, it was silent. Anyway, after fixing that, the errors went away for me.