Open ma-pony opened 1 year ago
Encountering the same issue here.
2023-05-24 09:10:35 [scrapy.core.engine] INFO: Spider closed (finished)
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing download handler
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing browser
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing download handler
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing browser
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-5112' coro=<_make_request_logger.<locals>._log_request() running at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/playwright/_impl/_connection.py:168]>
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-5113' coro=<_make_request_logger.<locals>._log_request() running at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/playwright/_impl/_connection.py:168]>
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
...[many more here]
Logs seem to point here (using scrapy-playwright 0.0.26
) but when I read this question I started wondering if this could be related too.
A nice starting point would be understanding if this is a scrapy-playwright
or playwright
kind of issue. What do you think?
This seems to be related to https://github.com/jfhbrook/pyee/issues/120. Keeping a reference to the tasks prevents this in my tests.
Please share a minimal, reproducible example.
Please share a minimal, reproducible example.
This was an occasional issue, the code that caused the problem was removed by me at the time, and i will try to reproduce the problem recently
Using the example from #233 I'm seeing much less of these messages after upgrading pyee from 9.0.4 to 11.0.0.
@elacuesta Any luck on a solution? I have a long-running scraper with this issue as well. It's not consistent and happens intermittently. Sometimes errors in 10 minutes, sometimes after 4 hours.
I just took your advice and upgraded my Pyee version to 11.0, I will try again.
@elacuesta Any luck on a solution?
I don't have plans to work on this, from what I see it's an upstream issue.
I'm getting a ton of these errors all of a sudden. I was running scrapy + playwright without issue for several months. Was this issue raised with Pyee?
For reference, my environment is using: python 3.11.6 scrapy 2.11.0 scrapy-playwright 0.0.33 twisted 22.10.0 pyee 11.0.1
Example logs:
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4511' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4514' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4512' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4515' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4763' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4766' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4764' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4767' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4772' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4773' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4797' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4798' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4809' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4812' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4810' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4813' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4863' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4864' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4869' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4872' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4875' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4870' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4873' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4878' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4881' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4879' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4882' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4884' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4887' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4885' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4888' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4876' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4931' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4932' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
@ThinksFast I was never able to solve the issue here. I ended up migrating to scrapy-splash.
@ThinksFast I was never able to solve the issue here. I ended up migrating to scrapy-splash.
Thanks for the follow up! I found out earlier today I was getting this issue due to an exception in my Scrapy pipeline file. The pipeline function to save the data was throwing an exception due to Unique Constraints set up on the table columns. But this exception wasn't getting logged for some reason, it was silent. Anyway, after fixing that, the errors went away for me.
I sometimes get this error when i use scrapy-pilaywright
(Edit) Reproducible example at #233