search

SpiderClub / haipproxy

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
https://spiderclub.github.io/haipproxy/
MIT License
5.44k stars 912 forks source link

运行时候报错：Unhandled error in Deferred: #69

Closed musicking closed 6 years ago

musicking commented 6 years ago

环境搭建好了，按照文档执行python crawler_booter.py --usage crawler 出现以下错误：

root@~#python crawler_booter.py --usage crawler Unhandled error in Deferred:

Unhandled error in Deferred:

Unhandled error in Deferred:

Unhandled error in Deferred:

Unhandled error in Deferred:

root@~#scrapy version -v

Scrapy : 1.5.0 lxml : 4.2.4.0 libxml2 : 2.9.8 cssselect : 1.0.3 parsel : 1.5.0 w3lib : 1.19.0 Twisted : 17.1.0 Python : 3.5.6 (default, Aug 8 2018, 18:36:31) - [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] pyOpenSSL : 18.0.0 (OpenSSL 1.1.0h 27 Mar 2018) cryptography : 2.3 Platform : Linux-2.6.32-504.el6.x86_64-x86_64-with-centos-6.6-Final

root@~#docker run -p 8050:8050 scrapinghub/splash

2018-08-09 03:18:26+0000 [-] Log opened. 2018-08-09 03:18:26.263532 [-] Splash version: 3.2 2018-08-09 03:18:26.264100 [-] Qt 5.9.1, PyQt 5.9, WebKit 602.1, sip 4.19.3, Twisted 16.1.1, Lua 5.2 2018-08-09 03:18:26.264174 [-] Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] 2018-08-09 03:18:26.264231 [-] Open files limit: 10240 2018-08-09 03:18:26.264281 [-] Can't bump open files limit 2018-08-09 03:18:26.367108 [-] Xvfb is started: ['Xvfb', ':2053627847', '-screen', '0', '1024x768x24', '-nolisten', 'tcp'] QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' 2018-08-09 03:18:26.442532 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles 2018-08-09 03:18:26.521230 [-] verbosity=1 2018-08-09 03:18:26.521334 [-] slots=50 2018-08-09 03:18:26.521397 [-] argument_cache_max_entries=500 2018-08-09 03:18:26.521640 [-] Web UI: enabled, Lua: enabled (sandbox: enabled) 2018-08-09 03:18:26.521711 [-] Server listening on 0.0.0.0:8050 2018-08-09 03:18:26.522198 [-] Site starting on 8050 2018-08-09 03:18:26.522284 [-] Starting factory <twisted.web.server.Site object at 0x7fb37add67b8>

Google了很多方案都解决不了，有知道怎么解决这个问题的吗？感谢~

ResolveWang commented 6 years ago

看看这个issue是否和你的符合

musicking commented 6 years ago

@ResolveWang 谢谢，我看了一下日志也是缺少sqlite3模块，应该搜一下issue再提问

musicking commented 6 years ago

参考这篇文章解决_sqlite3问题，感谢！ https://blog.csdn.net/zd147896325/article/details/80092563