nfe-w / aio-dynamic-push

一款整合多平台动态/直播开播提醒检测与推送的小工具, 目前支持:B站、微博、小红书、抖音、斗鱼、虎牙。
MIT License
125 stars 16 forks source link

小红书无法获取到动态,抖音查询任务报错 #23

Closed LionYu66 closed 2 months ago

LionYu66 commented 2 months ago

在6月24号正常推送一次后,突然开始无法获取作者的作品列表,报错如下:

stderr: 2024-06-24 17:53:50,324 - 8-MainThread - query_xhs.py[line: 85] - INFO: 【小红书-查询动态状态-任务_xhs】【repboy】上一条动态id[667819c5000000001c026e0f],本条动态id[66793707000000001f005f45] stderr: 2024-06-24 17:53:50,709 - 8-MainThread - query_xhs.py[line:102] - INFO: 【小红书-查询动态状态-任务_xhs】【repboy】动态有更新,准备推送:【修勾日记|给你们介绍下新朋友(蓝胖子)】#贝特恩奶粉[话题 stderr: 2024-06-24 17:53:52,965 - 8-MainThread - bark.py[line: 17] - INFO: 【推送_推送通道_Bark】成功 stderr: 2024-06-24 20:34:38,479 - 8-MainThread - util.py[line: 28] - ERROR: 【小红书-查询动态状态-任务_xhs】:HTTPSConnectionPool(host='www.xiaohongshu.com', port=443): Read timed out. (read timeout=10) stderr: Traceback (most recent call last): stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request stderr: self._validate_conn(conn) stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn stderr: conn.connect() stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect stderr: self.sock = ssl_wrapsocket( stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket stderr: ssl_sock = _ssl_wrap_socketimpl( stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl stderr: return ssl_context.wrap_socket(sock, server_hostname=server_hostname) stderr: File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket stderr: return self.sslsocket_class._create( stderr: File "/usr/local/lib/python3.9/ssl.py", line 1074, in _create stderr: self.do_handshake() stderr: File "/usr/local/lib/python3.9/ssl.py", line 1343, in do_handshake stderr: self._sslobj.do_handshake() stderr: socket.timeout: _ssl.c:1116: The handshake operation timed out stderr: stderr: During handling of the above exception, another exception occurred: stderr: stderr: Traceback (most recent call last): stderr: File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send stderr: resp = conn.urlopen( stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen stderr: retries = retries.increment( stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment stderr: raise six.reraise(type(error), error, _stacktrace) stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise stderr: raise value stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen stderr: httplib_response = self._make_request( stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 389, in _make_request stderr: self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) stderr: File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout stderr: raise ReadTimeoutError( stderr: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.xiaohongshu.com', port=443): Read timed out. (read timeout=10) stderr: stderr: During handling of the above exception, another exception occurred: stderr: stderr: Traceback (most recent call last): stderr: File "/app/common/util.py", line 26, in requests_get stderr: response = requests.get(url, headers=headers, params=params, proxies=proxies, timeout=10, verify=False) stderr: File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 73, in get stderr: return request("get", url, params=params, kwargs) stderr: File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request stderr: return session.request(method=method, url=url, kwargs) stderr: File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request stderr: resp = self.send(prep, send_kwargs) stderr: File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send stderr: r = adapter.send(request, kwargs) stderr: File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 532, in send stderr: raise ReadTimeout(e, request=request) stderr: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.xiaohongshu.com', port=443): Read timed out. (read timeout=10) stderr: 2024-06-25 09:56:59,216 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52 stderr: 2024-06-25 15:58:21,558 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52 stderr: 2024-06-25 20:49:26,920 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52 stderr: 2024-06-26 09:51:44,619 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52 stderr: 2024-06-26 20:44:12,589 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52

其后进行重启报错如下:

stderr: 2024-06-26 22:14:17,514 - 8-MainThread - config.py[line: 18] - INFO: 加载配置common: {'proxy_pool': {'enable': False, 'proxy_pool_url': 'http://192.168.100.1:5010'}} stderr: 2024-06-26 22:14:17,649 - 8-MainThread - config.py[line: 23] - INFO: 加载配置query_task: [{}] stderr: 2024-06-26 22:14:17,649 - 8-MainThread - config.py[line: 28] - INFO: 加载配置push_channel: [{}] stderr: 2024-06-26 22:14:17,649 - 8-MainThread - main.py[line: 12] - INFO: 开始初始化推送通道 stderr: 2024-06-26 22:14:17,649 - 8-MainThread - main.py[line: 19] - INFO: 初始化推送通道: 推送通道_Bark,通道类型: bark stderr: 2024-06-26 22:14:17,649 - 8-MainThread - main.py[line: 23] - INFO: 初始化查询任务 stderr: 2024-06-26 22:14:17,649 - 8-MainThread - main.py[line: 28] - INFO: 初始化查询任务: 任务_xhs,任务类型: xhs stderr: 2024-06-26 22:14:18,187 - 8-MainThread - query_xhs.py[line: 79] - INFO: 【小红书-查询动态状态-任务_xhs】【repboy】动态初始化:deque(['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', ''], maxlen=20) stderr: 2024-06-26 22:14:18,188 - 8-MainThread - main.py[line: 28] - INFO: 初始化查询任务: 任务_douyin,任务类型: douyin stderr: 2024-06-26 22:14:18,309 - 8-MainThread - query_douyin.py[line: 80] - INFO: 【抖音-生成ttwid-任务_douyin】成功:1%7CLjE84jNiWSa4HM379KEsl_bsmCKdWWJvf4YOYCdbpdA%7C1719411258%7Cd63e6bf866824583e18ee0f37f91c72e6fdaa998648a5e8a1420b61e666d151c stderr: 2024-06-26 22:14:18,309 - 8-MainThread - cache.py[line: 7] - INFO: [本地缓存]将: ttwid -> 1%7CLjE84jNiWSa4HM379KEsl_bsmCKdWWJvf4YOYCdbpdA%7C1719411258%7Cd63e6bf866824583e18ee0f37f91c72e6fdaa998648a5e8a1420b61e666d151c 存入缓存中 stderr: 2024-06-26 23:10:37,718 - 8-MainThread - config.py[line: 18] - INFO: 加载配置common: {'proxy_pool': {'enable': True, 'proxy_pool_url': 'http://192.168.100.1:5010'}} stderr: 2024-06-26 23:10:37,718 - 8-MainThread - proxy.py[line: 23] - INFO: 【ip池】已启用,地址: http://192.168.100.1:5010 stderr: 2024-06-26 23:10:37,853 - 8-MainThread - config.py[line: 23] - INFO: 加载配置query_task: [{}] stderr: 2024-06-26 23:10:37,853 - 8-MainThread - config.py[line: 28] - INFO: 加载配置push_channel: [{}] stderr: 2024-06-26 23:10:37,853 - 8-MainThread - main.py[line: 12] - INFO: 开始初始化推送通道 stderr: 2024-06-26 23:10:37,853 - 8-MainThread - main.py[line: 19] - INFO: 初始化推送通道: 推送通道_Bark,通道类型: bark stderr: 2024-06-26 23:10:37,853 - 8-MainThread - main.py[line: 23] - INFO: 初始化查询任务 stderr: 2024-06-26 23:10:37,853 - 8-MainThread - main.py[line: 28] - INFO: 初始化查询任务: 任务_xhs,任务类型: xhs stderr: 2024-06-26 23:10:47,430 - 8-MainThread - proxy.py[line: 64] - INFO: 【ip池】尝试多次均未获取到有效ip stderr: 2024-06-26 23:10:48,090 - 8-MainThread - query_xhs.py[line: 79] - INFO: 【小红书-查询动态状态-任务_xhs】【repboy】动态初始化:deque(['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', ''], maxlen=20) stderr: 2024-06-26 23:10:48,090 - 8-MainThread - main.py[line: 28] - INFO: 初始化查询任务: 任务_douyin,任务类型: douyin stderr: 2024-06-26 23:10:48,239 - 8-MainThread - query_douyin.py[line: 80] - INFO: 【抖音-生成ttwid-任务_douyin】成功:1%7Cj07X8-sbRT2KavzfC3EUMkg4YwjOFI4IQ6DFr_jcckg%7C1719414648%7C52fb5230857c141177a8e8e39cc3fbf60cdd9b696f69d5f59f0aec1e6bcd4fba stderr: 2024-06-26 23:10:48,239 - 8-MainThread - cache.py[line: 7] - INFO: [本地缓存]将: ttwid -> 1%7Cj07X8-sbRT2KavzfC3EUMkg4YwjOFI4IQ6DFr_jcckg%7C1719414648%7C52fb5230857c141177a8e8e39cc3fbf60cdd9b696f69d5f59f0aec1e6bcd4fba 存入缓存中 stderr: 2024-06-26 23:11:03,761 - 8-MainThread - proxy.py[line: 64] - INFO: 【ip池】尝试多次均未获取到有效ip stderr: 2024-06-26 23:11:03,957 - 8-MainThread - query_douyin.py[line: 41] - ERROR: 【抖音-查询任务-任务_douyin】出错:Expecting value: line 1 column 1 (char 0) stderr: Traceback (most recent call last): stderr: File "/app/query_task/query_douyin.py", line 36, in query stderr: self.query_dynamic(self.username_list[i], self.sec_uid_list[i]) stderr: File "/app/query_task/query_douyin.py", line 93, in query_dynamic stderr: result = json.loads(str(response.content, "utf-8")) stderr: File "/usr/local/lib/python3.9/json/init.py", line 346, in loads stderr: return _default_decoder.decode(s) stderr: File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode stderr: obj, end = self.raw_decode(s, idx=_w(s, 0).end()) stderr: File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode stderr: raise JSONDecodeError("Expecting value", s, err.value) from None stderr: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

LionYu66 commented 2 months ago

配置文件如下: common:

ip 代理池

proxy_pool:

是否启用 true/false

enable: false
# ip池地址,参考 https://github.com/jhao104/proxy_pool
proxy_pool_url: http://192.168.100.1:5010

query_task:

任务名称,可自定义

push_channel:

通道名称,唯一,可自定义不要纯数字,对应 query_task 中的 target_push_name_list

nfe-w commented 2 months ago

1、小红书频率过高会导致被ban一小会儿,你看日志,可以发现是好几小时才ban一次,所以这一点可以忽略不计:

stderr: 2024-06-25 09:56:59,216 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52
stderr: 2024-06-25 15:58:21,558 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52
stderr: 2024-06-25 20:49:26,920 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52
stderr: 2024-06-26 09:51:44,619 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52
stderr: 2024-06-26 20:44:12,589 - 8-MainThread - query_xhs.py[line: 52] - ERROR: 【小红书-查询动态状态-任务_xhs】请求返回数据为空,profile_id:61582f8f000000000201da52

2、抖音的视频更新检测现在挂了,我还不知道怎么修复,可以先关上(开播检测是正常的)

nfe-w commented 2 months ago

不对,我看错了,竟然是小红书动态的id变了,我看看代码

LionYu66 commented 2 months ago

辛苦了,顺便想问下以下这样配置是关闭抖音的视频更新检测吗? 然后时间如果想设置成今晚22:00到明天凌晨3:00该如何填写,之前我直接填写begin_time: "22:00",end_time: "03:00"似乎不能生效

nfe-w commented 2 months ago

1、唉,现在小红书的接口不返回笔记id了,先用标题判断是否变化吧(存在误判的可能),后面再研究研究,代码已提交; 2、enable_dynamic_check: false 是关闭视频更新检测的意思; 3、目前不支持跨日期的时间段

LionYu66 commented 2 months ago

好的辛苦作者了