Open luoshuijs opened 4 years ago
通过查看 https://billyzou0741326.github.io/bilibili-live-monitor-ts/#/faq/README 确认nofile上限已经设置最大,为65535
ss -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
或者
netstat -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
第一列是连接数 看下有多少
watch 'ss -t | awk '\'' $5 ~ /^[0-9]/ {print $5}'\'' | cut -d: -f1 | sort | uniq -c | sort -n'
top
指令看看有没有多个监控在跑吧 只跑一个的话应该不至于超上限...
top
查看发现只有一个监控在跑,通过pm2 logs
命令对其监视,发现
PM2 | App [live:0] exited with code [0] via signal [SIGINT]
PM2 | pid=17046 msg=process killed
PM2 | App [live:0] starting in -fork mode-
PM2 | App [live:0] online
0|live | [2020-03-10 17:12:48] WS server listening on 0.0.0.0:8999
0|live | [2020-03-10 17:12:48] WS server listening on 0.0.0.0:8998
0|live | [2020-03-10 17:12:48] TCP server listening on 0.0.0.0:8997
0|live | [2020-03-10 17:12:48] Http server listening on 0.0.0.0:9001
0|live | [2020-03-10 17:12:51] Setting up monitor @room 21981359 in 娱乐区
0|live | [2020-03-10 17:12:51] Setting up monitor @room 21093559 in 电台区
0|live | [2020-03-10 17:12:51] Setting up monitor @room 10139633 in 绘画区
0|live | [2020-03-10 17:12:51] Setting up monitor @room 5884993 in 手游区
0|live | [2020-03-10 17:12:51] Setting up monitor @room 293122 in 单机区
0|live | [2020-03-10 17:12:51] Setting up monitor @room 5468775 in 网游区
0|live | [2020-03-10 17:13:04] 2172290 @5101555 guard 舰长
0|live | [2020-03-10 17:13:56] 2172293 @8222458 guard 舰长
0|live | [2020-03-10 17:14:43] 2172295 @21513000 guard 舰长
0|live | [2020-03-10 17:14:52] Bilibili.getLiveCount - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:54] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live | [2020-03-10 17:14:55] Monitoring (静态) 77 + (动态) 15597
0|live | [Error: EMFILE: too many open files, open '/root/bilibili-live-monitor-ts/build/db/record.json'] {
0|live | errno: -24,
0|live | code: 'EMFILE',
0|live | syscall: 'open',
0|live | path: '/root/bilibili-live-monitor-ts/build/db/record.json'
0|live | }
^X^X0|live | [2020-03-10 17:15:30] 2172296 @146088 guard 舰长
1PM2 | Stopping app:live id:0
PM2 | pid=18205 msg=failed to kill - retrying in 100ms
PM2 | App [live:0] exited with code [0] via signal [SIGINT]
PM2 | pid=18205 msg=process killed
当CPU占用达到100%时执行ss -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
,发现连接数就卡在这里,后面没有任何增长
1014 120.92.78.57
1022 120.92.112.150
1034 120.92.158.137
重新运行几次都是这样
但是很奇怪的是,使用node ./build/main.js
(正常运行)却没有任何问题,几分钟后再次执行ss -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
1419 120.92.158.137
1455 120.92.144.250
1474 120.92.112.150
1485 120.92.78.57
一脸懵逼,再次修改此issue普通模式已经正常运行40分钟左右了 我™直接疑惑.jpg
看起来通过命令查看连接数,判断pm2模式运行下似乎是超出限制了,但是普通模式并没有出现这种情况,这很玄学了。
不知道是运行环境问题,而且我也发现这个机子似乎有点问题(?)
再次修改此issue已经再次确认/etc/security/limits.conf
并没问题内容如下
root soft nofile 65535
root hard nofile 65535
* soft nofile 65535
* hard nofile 65535
刚刚注意到一个文件夹/etc/security/limits.d
里面有个文件20-nproc.conf
刚刚通过网上查询到里面也有个限制
内容如下
* soft nproc 4096
root soft nproc unlimited
刚刚通过计算器计算得连接数正好接近4096,难道是这个配置文件的限制?但是问题是在top命令显示的时候已经以root权限运行,讲道理不应该出现这种情况。
刚刚修改了/etc/security/limits.d/20-nproc.conf
文件,修改后如下
* soft nproc 65535
root soft nproc unlimited
然后重启系统后(如果重新链接SSH不会生效),再次执行正常运行了5分钟,使用ss -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
查看链接数内容如下
2228 120.92.158.137
2260 120.92.144.250
2282 120.92.78.57
2309 120.92.112.150
所以就很奇怪了....讲道理pm2已经以root权限运行,top查看也是这样。应该遵循第二行的配置文件无限制才对= =
在这里我求求大佬优化一下异步操作吧,这样有点浪费系统资源啊 虽然没吃什么CPU但是讲道理哪怕是普通用户都用不着这么多 (运行20分钟后.....)
3762 120.92.158.137
3824 120.92.78.57
3871 120.92.144.250
3880 120.92.112.150
刚刚查看了这个文章:https://blog.csdn.net/gzliudan/article/details/51754701 (我看了好像就Centos有这覆盖之前配置的魔法操作) 里面说了守护进程的文件数量限制的修改,很详细 希望大佬在README更新一下,谢谢! (好了去写试卷了.jpg)
嗯 我看着改一下文档
有关连接数——
最开始写监控我的监听目标仅包括所有开播中的房间,当时的舰长覆盖率可能低至70% 后来在运行过程中累积静态房间(“永久”监听目标)以至现在95%以上应该
跑监控的话连接数是不可抗力 除非不要覆盖率2333
如题,但是普通运行下运行却没有任何问题。 在/root/.pm2/logs下有个错误日记,内容如下。 [Error: EMFILE: too many open files, open '/root/bilibili-live-monitor-ts/build/db/record.json'] { errno: -24, code: 'EMFILE', syscall: 'open', path: '/root/bilibili-live-monitor-ts/build/db/record.json' }