search

scrapy-plugins / scrapy-playwright

🎭 Playwright integration for Scrapy
BSD 3-Clause "New" or "Revised" License
1.02k stars 111 forks source link

Playwright Error :/ #125

Closed mecrayavcin closed 2 years ago

mecrayavcin commented 2 years ago

2022-09-10 20:45:30 [scrapy.utils.log] INFO: Scrapy 2.6.2 started (bot: pwspider) 2022-09-10 20:45:30 [scrapy.utils.log] INFO: Versions: lxml 4.9.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 2.0.1, Twisted 22.8.0, Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 22.0.0 (OpenSSL 3.0.5 5 Jul 2022), cryptography 38.0.1, Platform Windows-8.1-6.3.9600-SP0 2022-09-10 20:45:30 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'pwspider', 'NEWSPIDER_MODULE': 'pwspider.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['pwspider.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'} 2022-09-10 20:45:30 [asyncio] DEBUG: Using selector: SelectSelector 2022-09-10 20:45:30 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor 2022-09-10 20:45:30 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop 2022-09-10 20:45:30 [scrapy.extensions.telnet] INFO: Telnet Password: 55da5ff388d3eeb6 2022-09-10 20:45:30 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats'] 2022-09-10 20:45:30 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2022-09-10 20:45:30 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2022-09-10 20:45:30 [scrapy.middleware] INFO: Enabled item pipelines: [] 2022-09-10 20:45:30 [scrapy.core.engine] INFO: Spider opened 2022-09-10 20:45:31 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2022-09-10 20:45:31 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2022-09-10 20:45:31 [scrapy-playwright] INFO: Starting download handler 2022-09-10 20:45:31 [scrapy-playwright] INFO: Starting download handler 2022-09-10 20:45:31 [asyncio] ERROR: Task exception was never retrieved future: <Task finished name='Task-3' coro=<Connection.run() done, defined at C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_connection.py:217> exception=NotImplementedError()> Traceback (most recent call last): File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_connection.py", line 224, in run await self._transport.connect() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_transport.py", line 139, in connect raise exc File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_transport.py", line 127, in connect self._proc = await asyncio.create_subprocess_exec( File "C:\Users\Messi\anaconda3\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "C:\Users\Messi\anaconda3\lib\asyncio\base_events.py", line 1676, in subprocess_exec transport = await self._make_subprocess_transport( File "C:\Users\Messi\anaconda3\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport raise NotImplementedError NotImplementedError 2022-09-10 20:45:31 [asyncio] ERROR: Task exception was never retrieved future: <Task finished name='Task-4' coro=<Connection.run() done, defined at C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_connection.py:217> exception=NotImplementedError()> Traceback (most recent call last): File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_connection.py", line 224, in run await self._transport.connect() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_transport.py", line 139, in connect raise exc File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_transport.py", line 127, in connect self._proc = await asyncio.create_subprocess_exec( File "C:\Users\Messi\anaconda3\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "C:\Users\Messi\anaconda3\lib\asyncio\base_events.py", line 1676, in subprocess_exec transport = await self._make_subprocess_transport( File "C:\Users\Messi\anaconda3\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport raise NotImplementedError NotImplementedError 2022-09-10 20:45:31 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method ScrapyPlaywrightDownloadHandler._engine_started of <scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler object at 0x000000AEACB40E80>> Traceback (most recent call last): File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\internet\defer.py", line 1064, in adapt extracted = result.result() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\scrapy_playwright\handler.py", line 127, in _launch self.playwright = await self.playwright_context_manager.start() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\async_api\_context_manager.py", line 51, in start return await self.__aenter__() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\async_api\_context_manager.py", line 46, in __aenter__ playwright = AsyncPlaywright(next(iter(done)).result()) File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_connection.py", line 224, in run await self._transport.connect() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_transport.py", line 139, in connect raise exc File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_transport.py", line 127, in connect self._proc = await asyncio.create_subprocess_exec( File "C:\Users\Messi\anaconda3\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "C:\Users\Messi\anaconda3\lib\asyncio\base_events.py", line 1676, in subprocess_exec transport = await self._make_subprocess_transport( File "C:\Users\Messi\anaconda3\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport raise NotImplementedError NotImplementedError 2022-09-10 20:45:31 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method ScrapyPlaywrightDownloadHandler._engine_started of <scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler object at 0x000000AEACED5AC0>> Traceback (most recent call last): File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\internet\defer.py", line 1064, in adapt extracted = result.result() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\scrapy_playwright\handler.py", line 127, in _launch self.playwright = await self.playwright_context_manager.start() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\async_api\_context_manager.py", line 51, in start return await self.__aenter__() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\async_api\_context_manager.py", line 46, in __aenter__ playwright = AsyncPlaywright(next(iter(done)).result()) File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_connection.py", line 224, in run await self._transport.connect() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_transport.py", line 139, in connect raise exc File "C:\Users\Messi\Desktop\Bot\lib\site-packages\playwright\_impl\_transport.py", line 127, in connect self._proc = await asyncio.create_subprocess_exec( File "C:\Users\Messi\anaconda3\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "C:\Users\Messi\anaconda3\lib\asyncio\base_events.py", line 1676, in subprocess_exec transport = await self._make_subprocess_transport( File "C:\Users\Messi\anaconda3\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport raise NotImplementedError NotImplementedError 2022-09-10 20:45:36 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://arsiv.mackolik.com/robots.txt> (referer: None) 2022-09-10 20:45:36 [protego] DEBUG: Rule at line 1 without any user agent to enforce it on. 2022-09-10 20:45:36 [protego] DEBUG: Rule at line 2 without any user agent to enforce it on. 2022-09-10 20:45:36 [protego] DEBUG: Rule at line 8 without any user agent to enforce it on. 2022-09-10 20:45:36 [protego] DEBUG: Rule at line 9 without any user agent to enforce it on. 2022-09-10 20:45:36 [protego] DEBUG: Rule at line 10 without any user agent to enforce it on. 2022-09-10 20:45:36 [protego] DEBUG: Rule at line 11 without any user agent to enforce it on. 2022-09-10 20:45:36 [protego] DEBUG: Rule at line 12 without any user agent to enforce it on. 2022-09-10 20:45:36 [scrapy.core.scraper] ERROR: Error downloading <GET https://arsiv.mackolik.com/Puan-Durumu/s=61643/Turkiye-Super-Lig> Traceback (most recent call last): File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\internet\defer.py", line 1692, in _inlineCallbacks result = context.run( File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\python\failure.py", line 518, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "C:\Users\Messi\Desktop\Bot\lib\site-packages\scrapy\core\downloader\middleware.py", line 49, in process_request return (yield download_func(request=request, spider=spider)) File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\internet\defer.py", line 1064, in adapt extracted = result.result() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\scrapy_playwright\handler.py", line 247, in _download_request page = await self._create_page(request) File "C:\Users\Messi\Desktop\Bot\lib\site-packages\scrapy_playwright\handler.py", line 185, in _create_page context = await self._create_browser_context( File "C:\Users\Messi\Desktop\Bot\lib\site-packages\scrapy_playwright\handler.py", line 160, in _create_browser_context await self._maybe_launch_browser() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\scrapy_playwright\handler.py", line 144, in _maybe_launch_browser logger.info(f"Launching browser {self.browser_type.name}") AttributeError: 'ScrapyPlaywrightDownloadHandler' object has no attribute 'browser_type' 2022-09-10 20:45:36 [scrapy.core.engine] INFO: Closing spider (finished) 2022-09-10 20:45:36 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/exception_count': 1, 'downloader/exception_type_count/builtins.AttributeError': 1, 'downloader/request_bytes': 483, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 930, 'downloader/response_count': 1, 'downloader/response_status_count/404': 1, 'elapsed_time_seconds': 5.366164, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2022, 9, 10, 17, 45, 36, 593375), 'httpcompression/response_bytes': 1245, 'httpcompression/response_count': 1, 'log_count/DEBUG': 11, 'log_count/ERROR': 5, 'log_count/INFO': 12, 'response_received_count': 1, 'robotstxt/request_count': 1, 'robotstxt/response_count': 1, 'robotstxt/response_status_count/404': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'start_time': datetime.datetime(2022, 9, 10, 17, 45, 31, 227211)} 2022-09-10 20:45:36 [scrapy.core.engine] INFO: Spider closed (finished) 2022-09-10 20:45:36 [scrapy-playwright] INFO: Closing download handler 2022-09-10 20:45:36 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method DownloadHandlers._close of <scrapy.core.downloader.handlers.DownloadHandlers object at 0x000000AEACB27CA0>> Traceback (most recent call last): File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\internet\defer.py", line 1692, in _inlineCallbacks result = context.run( File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\python\failure.py", line 518, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "C:\Users\Messi\Desktop\Bot\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 81, in _close yield dh.close() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\internet\defer.py", line 1692, in _inlineCallbacks result = context.run( File "C:\Users\Messi\Desktop\Bot\lib\site-packages\twisted\python\failure.py", line 518, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "C:\Users\Messi\Desktop\Bot\lib\site-packages\sc ose yield deferred_from_coro(self._close()) File "C:\Users\Messi\Desktop\Bot\lib\site-packages\tw t extracted = result.result() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\sc lose await self.playwright_context_manager.__aexit__() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\pl e 54, in __aexit__ await self._connection.stop_async() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\pl stop_async self._transport.request_stop() File "C:\Users\Messi\Desktop\Bot\lib\site-packages\pl request_stop assert self._output AttributeError: 'PipeTransport' object has no attribute

I did this tutorial https://www.youtube.com/watch?v=0wO7K-SoUHM&t=487s&ab_channel=JohnWatsonRooney for this site https://arsiv.mackolik.com/Puan-Durumu/s=61643/Turkiye-Super-Lig

import scrapy

class MackolikSpider(scrapy.Spider):
    name = 'mackolik'

    def start_requests(self):
        yield scrapy.Request('https://arsiv.mackolik.com/Puan-Durumu/s=61643/Turkiye-Super-Lig', meta={'playwright': True})

    def parse(self, response):
        yield{
            'text': response.text
        }

Here is my spider

What is the problem? Thanks very much.

elacuesta commented 2 years ago

scrapy-playwright does not work on Windows (#7). See more details at https://github.com/scrapy-plugins/scrapy-playwright#known-issues.