search

crawlab-team / crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
https://www.crawlab.cn
BSD 3-Clause "New" or "Revised" License
11.38k stars 1.8k forks source link

AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD #1328

Closed AslanLau closed 1 year ago

AslanLau commented 1 year ago

Describe the bug 按照文档创建scrapy_quotes爬虫，上传后运行，报 “AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'”，粗略查了一下，sslv3已经被废弃，最新的openssl版本已经到了23.2.0。 其他的报错咱也看不懂，请问大佬这种情况如何处理？提前致谢~

以下是日志： ` 2023-06-14 19:37:17 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "http" Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/init.py", line 49, in _load_handler dhcls = load_object(path) File "/usr/local/lib/python3.10/dist-packages/scrapy/utils/misc.py", line 61, in load_object mod = import_module(module) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/http.py", line 2, in from scrapy.core.downloader.handlers.http11 import ( File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/http11.py", line 23, in from scrapy.core.downloader.contextfactory import load_context_factory_from_settings File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/contextfactory.py", line 11, in from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/tls.py", line 23, in METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended) AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD' 2023-06-14 19:37:17 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "https" Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/init.py", line 49, in _load_handler dhcls = load_object(path) File "/usr/local/lib/python3.10/dist-packages/scrapy/utils/misc.py", line 61, in load_object mod = import_module(module) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/http.py", line 2, in from scrapy.core.downloader.handlers.http11 import ( File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/http11.py", line 23, in from scrapy.core.downloader.contextfactory import load_context_factory_from_settings File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/contextfactory.py", line 11, in from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/tls.py", line 23, in METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended) AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD' 2023-06-14 19:37:17 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.s3.S3DownloadHandler" for scheme "s3" Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/init.py", line 49, in _load_handler dhcls = load_object(path) File "/usr/local/lib/python3.10/dist-packages/scrapy/utils/misc.py", line 61, in load_object mod = import_module(module) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/s3.py", line 1, in from scrapy.core.downloader.handlers.http import HTTPDownloadHandler File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/http.py", line 2, in from scrapy.core.downloader.handlers.http11 import ( File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/http11.py", line 23, in from scrapy.core.downloader.contextfactory import load_context_factory_from_settings File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/contextfactory.py", line 11, in from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/tls.py", line 23, in METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended) AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD' Unhandled error in Deferred: 2023-06-14 19:37:17 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/scrapy/crawler.py", line 206, in crawl return self._crawl(crawler, *args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/scrapy/crawler.py", line 210, in _crawl d = crawler.crawl(args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/twisted/internet/defer.py", line 1947, in unwindGenerator return _cancellableInlineCallbacks(gen) File "/usr/local/lib/python3.10/dist-packages/twisted/internet/defer.py", line 1857, in _cancellableInlineCallbacks _inlineCallbacks(None, gen, status, _copy_context()) --- --- File "/usr/local/lib/python3.10/dist-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks result = context.run(gen.send, result) File "/usr/local/lib/python3.10/dist-packages/scrapy/crawler.py", line 102, in crawl self.engine = self._create_engine() File "/usr/local/lib/python3.10/dist-packages/scrapy/crawler.py", line 116, in _createengine return ExecutionEngine(self, lambda : self.stop()) File "/usr/local/lib/python3.10/dist-packages/scrapy/core/engine.py", line 83, in init self.downloader = downloader_cls(crawler) File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/init.py", line 83, in init self.middleware = DownloaderMiddlewareManager.from_crawler(crawler) File "/usr/local/lib/python3.10/dist-packages/scrapy/middleware.py", line 59, in from_crawler return cls.from_settings(crawler.settings, crawler) File "/usr/local/lib/python3.10/dist-packages/scrapy/middleware.py", line 40, in from_settings mwcls = load_object(clspath) File "/usr/local/lib/python3.10/dist-packages/scrapy/utils/misc.py", line 61, in load_object mod = import_module(module) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import

File "", line 1027, in _find_and_load

File "", line 1006, in _find_and_load_unlocked

File "", line 688, in _load_unlocked

File "", line 883, in exec_module

File "", line 241, in _call_with_frames_removed

File "/usr/local/lib/python3.10/dist-packages/scrapy/downloadermiddlewares/retry.py", line 27, in from scrapy.core.downloader.handlers.http11 import TunnelError File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/http11.py", line 23, in from scrapy.core.downloader.contextfactory import load_context_factory_from_settings File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/contextfactory.py", line 11, in from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/tls.py", line 23, in METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended) builtins.AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'

2023-06-14 19:37:17 [twisted] CRITICAL: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks result = context.run(gen.send, result) File "/usr/local/lib/python3.10/dist-packages/scrapy/crawler.py", line 102, in crawl self.engine = self._create_engine() File "/usr/local/lib/python3.10/dist-packages/scrapy/crawler.py", line 116, in _createengine return ExecutionEngine(self, lambda : self.stop()) File "/usr/local/lib/python3.10/dist-packages/scrapy/core/engine.py", line 83, in init self.downloader = downloader_cls(crawler) File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/init.py", line 83, in init self.middleware = DownloaderMiddlewareManager.from_crawler(crawler) File "/usr/local/lib/python3.10/dist-packages/scrapy/middleware.py", line 59, in from_crawler return cls.from_settings(crawler.settings, crawler) File "/usr/local/lib/python3.10/dist-packages/scrapy/middleware.py", line 40, in from_settings mwcls = load_object(clspath) File "/usr/local/lib/python3.10/dist-packages/scrapy/utils/misc.py", line 61, in load_object mod = import_module(module) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/usr/local/lib/python3.10/dist-packages/scrapy/downloadermiddlewares/retry.py", line 27, in from scrapy.core.downloader.handlers.http11 import TunnelError File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/handlers/http11.py", line 23, in from scrapy.core.downloader.contextfactory import load_context_factory_from_settings File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/contextfactory.py", line 11, in from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions File "/usr/local/lib/python3.10/dist-packages/scrapy/core/downloader/tls.py", line 23, in METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended) AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD' `

AslanLau commented 1 year ago

修正一下，是pyOpenSSL的版本为23.2.0

tikazyq commented 1 year ago

升级scrapy到最新版本应该可以修复这个问题 ---- Replied Message ---- | From | wonder @.> | | Date | 06/15/2023 09:55 | | To | crawlab-team/crawlab @.> | | Cc | Subscribed @.***> | | Subject | Re: [crawlab-team/crawlab] AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD (Issue #1328) |

修正一下，是pyOpenSSL的版本为23.2.0

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

tikazyq commented 1 year ago

Latest build https://github.com/crawlab-team/crawlab/actions/runs/5274965719 should fix this issue