search

ice-tong / TumblrSpider

使用scrapy编写的python爬虫
68 stars 11 forks source link

vps上跑出现问题，盼解答 #3

Open shmilyoo opened 5 years ago

shmilyoo commented 5 years ago

2018-12-04 11:05:45 [scrapy] INFO: Scrapy 1.0.3 started (bot: tumblrSpider) 2018-12-04 11:05:45 [scrapy] INFO: Optional features available: ssl, http11, boto 2018-12-04 11:05:45 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tumblrSpider.spiders', 'SPIDER_MODULES': ['tumblrSpider.spiders'], 'BOT_NAME': 'tumblrSpider'} 2018-12-04 11:05:45 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState 2018-12-04 11:05:45 [boto] DEBUG: Retrieving credentials from metadata server. 2018-12-04 11:05:45 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, LocalProxySpiderMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 2018-12-04 11:05:45 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 2018-12-04 11:05:45 [scrapy] INFO: Enabled item pipelines: TumblrspiderPipeline 2018-12-04 11:05:45 [scrapy] INFO: Spider opened 2018-12-04 11:05:45 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2018-12-04 11:05:45 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 myusername 2018-12-04 11:05:46 [scrapy] ERROR: Error downloading <GET https://myusername.tumblr.com/api/read/json?start=0&num=200> Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 555, in connect timeout=self._timeout, bindAddress=self._bindAddress) File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 482, in connectTCP c = tcp.Connector(host, port, factory, timeout, bindAddress, self) File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 1165, in init if abstract.isIPv6Address(host): File "/usr/lib/python2.7/dist-packages/twisted/internet/abstract.py", line 522, in isIPv6Address if '%' in addr: TypeError: argument of type 'NoneType' is not iterable 2018-12-04 11:05:46 [scrapy] INFO: Closing spider (finished) 2018-12-04 11:05:46 [scrapy] INFO: Dumping Scrapy stats: {'downloader/exception_count': 1, 'downloader/exception_type_count/exceptions.TypeError': 1, 'downloader/request_bytes': 513, 'downloader/request_count': 1, 'downloader/request_method_count/GET': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2018, 12, 4, 11, 5, 46, 172847), 'log_count/DEBUG': 2, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'start_time': datetime.datetime(2018, 12, 4, 11, 5, 45, 960112)} 2018-12-04 11:05:46 [scrapy] INFO: Spider closed (finished)

项目有三个tumblrSpider目录，是在最深的那个目录执行么？ 使用python2还是3，如果在国外vps运行，不需要代理的话要怎么配置，我只修改了初始地址和深度为2，跑下来出错，python2. 好久没用python了，看得头晕😵。谢谢

shmilyoo commented 5 years ago

把setting中的中间件注释后，还是出错

2018-12-04 11:13:38 [scrapy] INFO: Scrapy 1.0.3 started (bot: tumblrSpider) 2018-12-04 11:13:38 [scrapy] INFO: Optional features available: ssl, http11, boto 2018-12-04 11:13:38 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tumblrSpider.spiders', 'SPIDER_MODULES': ['tumblrSpider.spiders'], 'BOT_NAME': 'tumblrSpider'} 2018-12-04 11:13:38 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState 2018-12-04 11:13:38 [boto] DEBUG: Retrieving credentials from metadata server. 2018-12-04 11:13:38 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 2018-12-04 11:13:38 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 2018-12-04 11:13:38 [scrapy] INFO: Enabled item pipelines: TumblrspiderPipeline 2018-12-04 11:13:38 [scrapy] INFO: Spider opened 2018-12-04 11:13:38 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2018-12-04 11:13:38 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 myusername 2018-12-04 11:13:40 [scrapy] DEBUG: Crawled (200) <GET https://myusername.tumblr.com/api/read/json?start=0&num=200> (referer: None) 2018-12-04 11:13:40 [scrapy] ERROR: Spider error processing <GET https://myusername.tumblr.com/api/read/json?start=0&num=200> (referer: None) Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback yield next(it) File "/usr/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py", line 28, in process_spider_output for x in result: File "/usr/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py", line 22, in return (_set_referer(r) for r in result or ()) File "/usr/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in return (r for r in result or () if _filter(r)) File "/usr/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result or () if _filter(r)) File "/opt/tumblr/TumblrSpider/tumblrSpider/tumblrSpider/spiders/tbr.py", line 31, in parse data = response.text[22:-2] AttributeError: 'TextResponse' object has no attribute 'text' 2018-12-04 11:13:40 [scrapy] INFO: Closing spider (finished) 2018-12-04 11:13:40 [scrapy] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 513, 'downloader/request_count': 1, 'downloader/request_method_count/GET': 1, 'downloader/response_bytes': 635441, 'downloader/response_count': 1, 'downloader/response_status_count/200': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2018, 12, 4, 11, 13, 40, 788807), 'log_count/DEBUG': 3, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'response_received_count': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'spider_exceptions/AttributeError': 1, 'start_time': datetime.datetime(2018, 12, 4, 11, 13, 38, 538111)} 2018-12-04 11:13:40 [scrapy] INFO: Spider closed (finished)

ice-tong commented 5 years ago

代码是python3，我也不确定你这报错是不是因为python版本不同的问题，换成python3试试。