search

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://scrapy.org
BSD 3-Clause "New" or "Revised" License
51.16k stars 10.35k forks source link

服务器部署遇到问题 #6339

Closed Hao1617 closed 1 week ago

Hao1617 commented 1 week ago

Description

[部署时运行 遇到 2024-05-06 14:24:08 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "http" ]

2024-05-06 14:24:08 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: douyin_spider)
2024-05-06 14:24:08 [scrapy.utils.log] INFO: Versions: lxml 5.2.1.0, libxml2 2.12.6, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.4.0, Python 3.7.0 (default, Apr 20 2022, 17:56:38) - [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)], pyOpenSSL 24.1.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.6, Platform Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
2024-05-06 14:24:08 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'douyin_spider',
 'DOWNLOAD_DELAY': 1,
 'DUPEFILTER_CLASS': 'scrapy_redis.dupefilter.RFPDupeFilter',
 'NEWSPIDER_MODULE': 'douyin_spider.spiders',
 'SCHEDULER': 'scrapy_redis.scheduler.Scheduler',
 'SPIDER_MODULES': ['douyin_spider.spiders']}
2024-05-06 14:24:08 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2024-05-06 14:24:08 [scrapy.extensions.telnet] INFO: Telnet Password: 9860e142f9e01641
2024-05-06 14:24:08 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2024-05-06 14:24:08 [searchItem] INFO: Reading start URLs from redis key 'py21' (batch size: 16, encoding: utf-8)
2024-05-06 14:24:08 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "http"
Traceback (most recent call last):
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/__init__.py", line 49, in _load_handler
    dhcls = load_object(path)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/utils/misc.py", line 61, in load_object
    mod = import_module(module)
  File "/usr/local/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http.py", line 2, in <module>
    from scrapy.core.downloader.handlers.http11 import (
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 23, in <module>
    from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/contextfactory.py", line 11, in <module>
    from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/tls.py", line 23, in <module>
    METHOD_SSLv3: SSL.SSLv3_METHOD,                     # SSL 3 (NOT recommended)
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
2024-05-06 14:24:09 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "https"
Traceback (most recent call last):
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/__init__.py", line 49, in _load_handler
    dhcls = load_object(path)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/utils/misc.py", line 61, in load_object
    mod = import_module(module)
  File "/usr/local/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http.py", line 2, in <module>
    from scrapy.core.downloader.handlers.http11 import (
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 23, in <module>
    from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/contextfactory.py", line 11, in <module>
    from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/tls.py", line 23, in <module>
    METHOD_SSLv3: SSL.SSLv3_METHOD,                     # SSL 3 (NOT recommended)
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
2024-05-06 14:24:09 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.s3.S3DownloadHandler" for scheme "s3"
Traceback (most recent call last):
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/__init__.py", line 49, in _load_handler
    dhcls = load_object(path)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/utils/misc.py", line 61, in load_object
    mod = import_module(module)
  File "/usr/local/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/s3.py", line 1, in <module>
    from scrapy.core.downloader.handlers.http import HTTPDownloadHandler
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http.py", line 2, in <module>
    from scrapy.core.downloader.handlers.http11 import (
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 23, in <module>
    from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/contextfactory.py", line 11, in <module>
    from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/tls.py", line 23, in <module>
    METHOD_SSLv3: SSL.SSLv3_METHOD,                     # SSL 3 (NOT recommended)
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
Unhandled error in Deferred:
2024-05-06 14:24:09 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 206, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 210, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/usr/local/python3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/usr/local/python3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "/usr/local/python3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 102, in crawl
    self.engine = self._create_engine()
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 116, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/engine.py", line 83, in __init__
    self.downloader = downloader_cls(crawler)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/__init__.py", line 83, in __init__
    self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/middleware.py", line 59, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/middleware.py", line 40, in from_settings
    mwcls = load_object(clspath)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/utils/misc.py", line 61, in load_object
    mod = import_module(module)
  File "/usr/local/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import

  File "<frozen importlib._bootstrap>", line 983, in _find_and_load

  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 728, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/downloadermiddlewares/retry.py", line 27, in <module>
    from scrapy.core.downloader.handlers.http11 import TunnelError
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 23, in <module>
    from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/contextfactory.py", line 11, in <module>
    from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/tls.py", line 23, in <module>
    METHOD_SSLv3: SSL.SSLv3_METHOD,                     # SSL 3 (NOT recommended)
builtins.AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'

2024-05-06 14:24:09 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "/usr/local/python3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 102, in crawl
    self.engine = self._create_engine()
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 116, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/engine.py", line 83, in __init__
    self.downloader = downloader_cls(crawler)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/__init__.py", line 83, in __init__
    self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/middleware.py", line 59, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/middleware.py", line 40, in from_settings
    mwcls = load_object(clspath)
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/utils/misc.py", line 61, in load_object
    mod = import_module(module)
  File "/usr/local/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/downloadermiddlewares/retry.py", line 27, in <module>
    from scrapy.core.downloader.handlers.http11 import TunnelError
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 23, in <module>
    from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/contextfactory.py", line 11, in <module>
    from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
  File "/usr/local/python3.7/lib/python3.7/site-packages/scrapy/core/downloader/tls.py", line 23, in <module>
    METHOD_SSLv3: SSL.SSLv3_METHOD,                     # SSL 3 (NOT recommended)
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
[root@VM-16-14-centos douyin_spider]# python3 version
(null): can't open file 'version': [Errno 2] No such file or directory

image

Versions

python 3.7.0 scrapy 2.6.1

Additional context

Any additional information, configuration, data or output from commands that might be necessary to reproduce or understand the issue. Please try not to include screenshots of code or the command line, paste the contents as text instead. You can use GitHub Flavored Markdown to make the text look better.

Hao1617 commented 1 week ago

How to fix it

wRAR commented 1 week ago

Please ask questions about your code on suitable platforms: https://scrapy.org/community/