jhao104 / proxy_pool

Python ProxyPool for web spider
https://jhao104.github.io/proxy_pool/
MIT License
21.53k stars 5.18k forks source link

ERROR redis connection error #705

Open tcotu opened 2 years ago

tcotu commented 2 years ago

2022-10-10 22:59:23,812 redisClient.py[line:149] ERROR redis connection error: Error while reading from 127.0.0.1:6379 : (104, 'Connection reset by peer') Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 824, in read_response response = self._parser.read_response(disable_decoding=disable_decoding) File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 318, in read_response raw = self._buffer.readline() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 249, in readline self._read_from_socket() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 192, in _read_from_socket data = self._sock.recv(socket_read_size) ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/proxy_pool/db/redisClient.py", line 144, in test self.getCount() File "/root/proxy_pool/db/redisClient.py", line 130, in getCount proxies = self.getAll(https=False) File "/root/proxy_pool/db/redisClient.py", line 112, in getAll items = self.__conn.hvals(self.name) File "/usr/local/lib/python3.6/site-packages/redis/commands/core.py", line 4887, in hvals return self.execute_command("HVALS", name) File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1235, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 1582, in get_connection connection.connect() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 623, in connect self.on_connect() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 713, in on_connect auth_response = self.read_response() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 830, in read_response raise ConnectionError(f"Error while reading from {hosterr}" f" : {e.args}") redis.exceptions.ConnectionError: Error while reading from 127.0.0.1:6379 : (104, 'Connection reset by peer') 2022-10-10 22:59:23,827 launcher.py[line:39] INFO exit! 2022-10-10 22:59:23,816 redisClient.py[line:149] ERROR redis connection error: Error while reading from 127.0.0.1:6379 : (104, 'Connection reset by peer') Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 824, in read_response response = self._parser.read_response(disable_decoding=disable_decoding) File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 318, in read_response raw = self._buffer.readline() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 249, in readline self._read_from_socket() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 192, in _read_from_socket data = self._sock.recv(socket_read_size) ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/proxy_pool/db/redisClient.py", line 144, in test self.getCount() File "/root/proxy_pool/db/redisClient.py", line 130, in getCount proxies = self.getAll(https=False) File "/root/proxy_pool/db/redisClient.py", line 112, in getAll items = self.__conn.hvals(self.name) File "/usr/local/lib/python3.6/site-packages/redis/commands/core.py", line 4887, in hvals return self.execute_command("HVALS", name) File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1235, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 1582, in get_connection connection.connect() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 623, in connect self.on_connect() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 713, in on_connect auth_response = self.read_response() File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 830, in read_response raise ConnectionError(f"Error while reading from {hosterr}" f" : {e.args}") redis.exceptions.ConnectionError: Error while reading from 127.0.0.1:6379 : (104, 'Connection reset by peer') 2022-10-10 22:59:23,842 launcher.py[line:39] INFO exit!

tcotu commented 2 years ago

i don't know why occur this,about redis connection.i may handle redis to trouble. hope to help me.

tcotu commented 2 years ago

i try to directly using redis success to descrip this problem

jhao104 commented 2 years ago

If you run with docker, you should use bridge networks . Otherwise, check network telnet 127.0.0.1 6379

TD-ah404 commented 2 years ago

如果使用 Docker 运行,则应使用 网桥网络 。否则,请检查网络telnet 127.0.0.1 6379

师傅,https一个ip都没有获取

todaygood commented 2 years ago

I met the same issue, the result of telnet 127.0.0.1 6379 is 'connected'

todaygood commented 2 years ago

因为一个docker容器是一个net namespace , 访问127.0.0.1:6379 时 127.0.0.1 指向proxy_pool 这个容器, 不是host,所以出错。

估计你的redis 也是docker启动的, 如果是,那启动proxy_pool docker时 加上参数--link redis(redis容器名字) 即可。

docker run -idt --name proxy_pool --link redis --env DB_CONN=redis://redis:6379/0 -p 5010:5010 jhao104/proxy_pool

如果你的redis 在host 上是原生二进制进程跑的, 那么启动proxy_pool docker时 加上参数

docker run -idt --name proxy_pool --network host --env DB_CONN=redis://127.0.0.1:6379/0 jhao104/proxy_pool (没测试,应该可以)

todaygood commented 2 years ago

建议使用docker-compose, 将proxy_pool 和redis 一起管理起来, 有两个好处:

  1. 默认会建一个network,可以使用容器名字, DB_CONN=redis://redis:6379/0 其中第redis是redis容器名字,避免使用127.0.0.1。
  2. 可以保证redis先于proxy_pool启动。