Closed vdusek closed 8 months ago
2023-12-15T15:10:18.262Z ACTOR: Pulling Docker image of build r9WYYLPAanPEMEmN2 from repository. 2023-12-15T15:10:18.498Z ACTOR: Creating Docker container. 2023-12-15T15:10:18.601Z ACTOR: Starting Docker container. 2023-12-15T15:10:20.553Z [apify] INFO Initializing actor... 2023-12-15T15:10:20.556Z [apify] INFO System info ({"apify_sdk_version": "1.4.0", "apify_client_version": "1.6.0", "python_version": "3.11.6", "os": "linux"}) 2023-12-15T15:10:20.558Z [apify] INFO Actor is being executed... 2023-12-15T15:10:21.125Z [apify] INFO proxy_url: ParseResult(scheme='http', netloc='auto:*********@10.0.82.23:8011', path='', params='', query='', fragment='') 2023-12-15T15:10:21.128Z [apify] INFO proxy_settings = {'url': 'http://auto:*********@10.0.82.23:8011', 'username': 'auto', 'password': '*********'} 2023-12-15T15:10:21.218Z [scrapy.utils.log] INFO Scrapy 2.11.0 started (bot: titlebot) 2023-12-15T15:10:21.222Z [scrapy.utils.log] INFO Scrapy 2.11.0 started (bot: titlebot) ({"message": "Scrapy 2.11.0 started (bot: titlebot)"}) 2023-12-15T15:10:21.224Z [scrapy.utils.log] INFO Versions: lxml 4.9.3.0, libxml2 2.10.3, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.11.6 (main, Nov 29 2023, 04:47:02) [GCC 10.2.1 20210110], pyOpenSSL 23.3.0 (OpenSSL 3.1.4 24 Oct 2023), cryptography 41.0.7, Platform Linux-6.1.52-71.125.amzn2023.x86_64-x86_64-with-glibc2.31 2023-12-15T15:10:21.227Z [scrapy.utils.log] INFO Versions: lxml 4.9.3.0, libxml2 2.10.3, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.11.6 (main, Nov 29 2023, 04:47:02) [GCC 10.2.1 20210110], pyOpenSSL 23.3.0 (OpenSSL 3.1.4 24 Oct 2023), cryptography 41.0.7, Platform Linux-6.1.52-71.125.amzn2023.x86_64-x86_64-with-glibc2.31 ({"message": "Versions: lxml 4.9.3.0, libxml2 2.10.3, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.11.6 (main, Nov 29 2023, 04:47:02) [GCC 10.2.1 20210110], pyOpenSSL 23.3.0 (OpenSSL 3.1.4 24 Oct 2023), cryptography 41.0.7, Platform Linux-6.1.52-71.125.amzn2023.x86_64-x86_64-with-glibc2.31"}) 2023-12-15T15:10:21.229Z [scrapy.addons] INFO Enabled addons: 2023-12-15T15:10:21.237Z [] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>"}) 2023-12-15T15:10:21.242Z [scrapy.addons] INFO Enabled addons: 2023-12-15T15:10:21.245Z [] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>", "message": "Enabled addons:\n[]"}) 2023-12-15T15:10:21.251Z [scrapy.extensions.telnet] INFO Telnet Password: 0cf7f4b3360ba082 2023-12-15T15:10:21.254Z [scrapy.extensions.telnet] INFO Telnet Password: 0cf7f4b3360ba082 ({"message": "Telnet Password: 0cf7f4b3360ba082"}) 2023-12-15T15:10:21.333Z [scrapy.middleware] INFO Enabled extensions: 2023-12-15T15:10:21.336Z ['scrapy.extensions.corestats.CoreStats', 2023-12-15T15:10:21.338Z 'scrapy.extensions.telnet.TelnetConsole', 2023-12-15T15:10:21.341Z 'scrapy.extensions.memusage.MemoryUsage', 2023-12-15T15:10:21.344Z 'scrapy.extensions.logstats.LogStats'] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>"}) 2023-12-15T15:10:21.346Z [scrapy.middleware] INFO Enabled extensions: 2023-12-15T15:10:21.348Z ['scrapy.extensions.corestats.CoreStats', 2023-12-15T15:10:21.351Z 'scrapy.extensions.telnet.TelnetConsole', 2023-12-15T15:10:21.353Z 'scrapy.extensions.memusage.MemoryUsage', 2023-12-15T15:10:21.355Z 'scrapy.extensions.logstats.LogStats'] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>", "message": "Enabled extensions:\n['scrapy.extensions.corestats.CoreStats',\n 'scrapy.extensions.telnet.TelnetConsole',\n 'scrapy.extensions.memusage.MemoryUsage',\n 'scrapy.extensions.logstats.LogStats']"}) 2023-12-15T15:10:21.357Z [scrapy.crawler] INFO Overridden settings: 2023-12-15T15:10:21.360Z {'BOT_NAME': 'titlebot', 2023-12-15T15:10:21.362Z 'DEPTH_LIMIT': 1, 2023-12-15T15:10:21.364Z 'LOG_LEVEL': 'INFO', 2023-12-15T15:10:21.366Z 'NEWSPIDER_MODULE': 'src.spiders', 2023-12-15T15:10:21.368Z 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 2023-12-15T15:10:21.371Z 'ROBOTSTXT_OBEY': True, 2023-12-15T15:10:21.373Z 'SCHEDULER': 'apify.scrapy.scheduler.ApifyScheduler', 2023-12-15T15:10:21.376Z 'SPIDER_MODULES': ['src.spiders']} 2023-12-15T15:10:21.378Z [scrapy.crawler] INFO Overridden settings: 2023-12-15T15:10:21.381Z {'BOT_NAME': 'titlebot', 2023-12-15T15:10:21.383Z 'DEPTH_LIMIT': 1, 2023-12-15T15:10:21.386Z 'LOG_LEVEL': 'INFO', 2023-12-15T15:10:21.388Z 'NEWSPIDER_MODULE': 'src.spiders', 2023-12-15T15:10:21.391Z 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 2023-12-15T15:10:21.393Z 'ROBOTSTXT_OBEY': True, 2023-12-15T15:10:21.395Z 'SCHEDULER': 'apify.scrapy.scheduler.ApifyScheduler', 2023-12-15T15:10:21.398Z 'SPIDER_MODULES': ['src.spiders']} ({"message": "Overridden settings:\n{'BOT_NAME': 'titlebot',\n 'DEPTH_LIMIT': 1,\n 'LOG_LEVEL': 'INFO',\n 'NEWSPIDER_MODULE': 'src.spiders',\n 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',\n 'ROBOTSTXT_OBEY': True,\n 'SCHEDULER': 'apify.scrapy.scheduler.ApifyScheduler',\n 'SPIDER_MODULES': ['src.spiders']}"}) 2023-12-15T15:10:21.435Z [scrapy.middleware] INFO Enabled downloader middlewares: 2023-12-15T15:10:21.438Z ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 2023-12-15T15:10:21.441Z 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 2023-12-15T15:10:21.443Z 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 2023-12-15T15:10:21.446Z 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 2023-12-15T15:10:21.448Z 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 2023-12-15T15:10:21.451Z 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 2023-12-15T15:10:21.453Z 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 2023-12-15T15:10:21.456Z 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 2023-12-15T15:10:21.458Z 'scrapy.downloadermiddlewares.stats.DownloaderStats', 2023-12-15T15:10:21.460Z 'src.middlewares.ApifyHttpProxyMiddleware', 2023-12-15T15:10:21.462Z 'apify.scrapy.middlewares.ApifyRetryMiddleware'] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>"}) 2023-12-15T15:10:21.465Z [scrapy.middleware] INFO Enabled downloader middlewares: 2023-12-15T15:10:21.467Z ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 2023-12-15T15:10:21.469Z 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 2023-12-15T15:10:21.472Z 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 2023-12-15T15:10:21.474Z 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 2023-12-15T15:10:21.476Z 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 2023-12-15T15:10:21.479Z 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 2023-12-15T15:10:21.481Z 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 2023-12-15T15:10:21.484Z 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 2023-12-15T15:10:21.486Z 'scrapy.downloadermiddlewares.stats.DownloaderStats', 2023-12-15T15:10:21.489Z 'src.middlewares.ApifyHttpProxyMiddleware', 2023-12-15T15:10:21.492Z 'apify.scrapy.middlewares.ApifyRetryMiddleware'] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>", "message": "Enabled downloader middlewares:\n['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\n 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\n 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\n 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\n 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\n 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\n 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\n 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\n 'scrapy.downloadermiddlewares.stats.DownloaderStats',\n 'src.middlewares.ApifyHttpProxyMiddleware',\n 'apify.scrapy.middlewares.ApifyRetryMiddleware']"}) 2023-12-15T15:10:21.495Z [scrapy.middleware] INFO Enabled spider middlewares: 2023-12-15T15:10:21.497Z ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 2023-12-15T15:10:21.500Z 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 2023-12-15T15:10:21.502Z 'scrapy.spidermiddlewares.referer.RefererMiddleware', 2023-12-15T15:10:21.505Z 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 2023-12-15T15:10:21.507Z 'scrapy.spidermiddlewares.depth.DepthMiddleware'] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>"}) 2023-12-15T15:10:21.510Z [scrapy.middleware] INFO Enabled spider middlewares: 2023-12-15T15:10:21.514Z ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 2023-12-15T15:10:21.516Z 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 2023-12-15T15:10:21.519Z 'scrapy.spidermiddlewares.referer.RefererMiddleware', 2023-12-15T15:10:21.521Z 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 2023-12-15T15:10:21.523Z 'scrapy.spidermiddlewares.depth.DepthMiddleware'] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>", "message": "Enabled spider middlewares:\n['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\n 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\n 'scrapy.spidermiddlewares.referer.RefererMiddleware',\n 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\n 'scrapy.spidermiddlewares.depth.DepthMiddleware']"}) 2023-12-15T15:10:21.526Z [scrapy.middleware] INFO Enabled item pipelines: 2023-12-15T15:10:21.528Z ['src.pipelines.TitleItemPipeline', 2023-12-15T15:10:21.531Z 'apify.scrapy.pipelines.ActorDatasetPushPipeline'] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>"}) 2023-12-15T15:10:21.533Z [scrapy.middleware] INFO Enabled item pipelines: 2023-12-15T15:10:21.535Z ['src.pipelines.TitleItemPipeline', 2023-12-15T15:10:21.538Z 'apify.scrapy.pipelines.ActorDatasetPushPipeline'] ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>", "message": "Enabled item pipelines:\n['src.pipelines.TitleItemPipeline',\n 'apify.scrapy.pipelines.ActorDatasetPushPipeline']"}) 2023-12-15T15:10:21.540Z [scrapy.core.engine] INFO Spider opened ({"spider": "<TitleSpider 'title_spider' at 0x7f2e82e9b7d0>"}) 2023-12-15T15:10:21.542Z [scrapy.core.engine] INFO Spider opened ({"spider": "<TitleSpider 'title_spider' at 0x7f2e82e9b7d0>", "message": "Spider opened"}) 2023-12-15T15:10:21.545Z [scrapy.extensions.logstats] INFO Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) ({"spider": "<TitleSpider 'title_spider' at 0x7f2e82e9b7d0>"}) 2023-12-15T15:10:21.547Z [scrapy.extensions.logstats] INFO Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) ({"spider": "<TitleSpider 'title_spider' at 0x7f2e82e9b7d0>", "message": "Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)"}) 2023-12-15T15:10:21.550Z [twisted] INFO TelnetConsole starting on 6023 2023-12-15T15:10:21.552Z [twisted] INFO TelnetConsole starting on 6023 ({"message": "TelnetConsole starting on 6023"}) 2023-12-15T15:10:21.554Z [scrapy.extensions.telnet] INFO Telnet console listening on 127.0.0.1:6023 ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>"}) 2023-12-15T15:10:21.557Z [scrapy.extensions.telnet] INFO Telnet console listening on 127.0.0.1:6023 ({"crawler": "<scrapy.crawler.Crawler object at 0x7f2e846aee10>", "message": "Telnet console listening on 127.0.0.1:6023"}) 2023-12-15T15:10:22.709Z [title_spider] INFO TitleSpider is parsing <200 https://apify.com/>... ({"spider": "<TitleSpider 'title_spider' at 0x7f2e82e9b7d0>"}) ...
Description
Example of such log