Open Toifyx opened 4 years ago
我的nginx配置了两个代理,图片展示的为当前服务器predixy连接CLOSE_WAIT的数量。当我依次kill掉predixy的进程的时候。连接数量由1000->500 ->2 . (剩余的两个为其他机器的连接 )
我将集群redis换成单点redis。测试结果。并未出现CLOSE_WAIT的情况。
压测连接慢慢上来后,最大到992,压测结束后连接变成0
predix日志记录的什么
predix日志记录的什么
你好,我找到部分日志 有异常的如下:
2019-11-04 15:00:26.908966 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909023 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909089 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909134 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909134 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909184 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909223 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909269 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909272 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909327 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909361 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909378 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909514 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909522 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909574 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909779 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909797 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909815 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909833 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909852 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909872 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909887 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909907 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909920 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909942 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909955 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.910008 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911019 N Handler.cpp:303 h 2 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911389 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911420 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911442 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911480 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911486 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911524 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911524 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911572 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911652 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911763 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911573 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911844 N Handler.cpp:303 h 2 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911869 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
这看起来有点像ListenSocket.cpp,这个文件throw的异常,是否是在这里没有close连接。
/*
* predixy - A high performance and full features proxy for redis.
* Copyright (C) 2017 Joyield, Inc. <joyield.com@gmail.com>
* All rights reserved.
*/
#include <string.h>
#include "Util.h"
#include "ListenSocket.h"
ListenSocket::ListenSocket(const char* addr, int type, int protocol)
{
mClassType = ListenType;
strncpy(mAddr, addr, sizeof(mAddr));
sockaddr_storage saddr;
socklen_t len = sizeof(saddr);
getFirstAddr(addr, type, protocol, (sockaddr*)&saddr, &len);
sockaddr* in = (sockaddr*)&saddr;
int fd = Socket::socket(in->sa_family, type, protocol);
int flags = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &flags, sizeof(flags));
int ret = ::bind(fd, in, len);
if (ret != 0) {
int err = errno;
::close(fd);
errno = err;
Throw(BindFail, "socket bind to %s fail %s", addr, StrError());
}
attach(fd);
}
void ListenSocket::listen(int backlog)
{
int ret = ::listen(fd(), backlog);
if (ret != 0) {
Throw(ListenFail, "socket listen fail %s", StrError());
}
}
int ListenSocket::accept(sockaddr* addr, socklen_t* len)
{
while (true) {
int c = ::accept(fd(), addr, len);
if (c < 0) {
if (errno == EAGAIN || errno == EWOULDBLOCK) {
return -1;
} else if (errno == EINTR || errno == ECONNABORTED) {
continue;
} else if (errno == EMFILE || errno == ENFILE) {
Throw(TooManyOpenFiles, "socket accept fail %s", StrError());
}
Throw(AcceptFail, "socket accept fail %s", StrError());
}
return c;
}
return -1;
}
其他日志。这里就不贴出来了(看起来应该是正常的输出日志)
我也碰到了连接数过多的问题,
。这里就不贴出来了(看起来应该是
你好,这个问题你解决了么?
我也碰到了连接数过多的问题,
你好这个问题,你解决了么?
。这里就不贴出来了(看起来应该是
你好,这个问题你解决了么?
没有,换其他方式了。没有使用代理
socket accept fail Too many open files的的报错,大概是文件句柄不够 问题查看: 用ulimit -a查看当前的文件句柄限制(最大65535) 再看下进程占用的文件句柄数,是否达到了限制 lsof -p 代理predixy的pid | wc -l
解决办法: vim /etc/security/limits.conf
* soft nofile 4096
* hard nofile 4096
再重启下代理进程即可
由于 https://github.com/openresty/lua-resty-redis openresty/lua-resty-redis,不支持cluster模式,所以我用predixy代理。当我在压测nginx性能时,predixy出现大量CLOSE_WAITE请求未关闭情况。导致nginx连接不上predixy。有可能是我哪里配置不对嘛,或者predixy有bug。
predixy :1.0.5 release
cluster config:
/etc/sysctl.conf