search

diennea / carapaceproxy

A Distributed Java Reverse Proxy
Apache License 2.0
24 stars 8 forks source link

Carapace > Too many open files #414

Open hamadodene opened 1 year ago

hamadodene commented 1 year ago 
Feb 24, 2023 1:12:21 PM org.carapaceproxy.core.ProxyRequestsManager lambda$forward$14
SEVERE: Error proxying request for endpoint mn1-tomcat12.dna:30443; request: ProxyRequest(id=39673553, request=GET{uri=/nl/res/2808/newsletters/185/img/logofooter_telepass.png, connection=SimpleConnection{channel=[id: 0xc1e1183f, L:
/xxxxx:4089 - R:/xxxxx:42070]}}, response=GET{uri=/nl/res/xxx/newsletters/xx/img/logofooter_telepass.png, connection=SimpleConnection{channel=[id: 0xc1e1183f, L:/10.219.105.142:4089 - R:/94.34.183.208:42070]}}, li
stener=HostPort[host=0.0.0.0, port=4089], action=MapResult(host=mn1-tomcat12.dna, port=30443, action=CACHE, routeId=secure_request, errorCode=0, resource=null, customHeaders=[CustomHeader(id=mapper-debug, name=X-Proxy-Path, value=se
cure_request;fehttps_a;fehttps_d;fehttps12, mode=add)], redirectLocation=null, redirectProto=null, redirectPath=null), userId=null, sessionId=null, startTs=1677240741324, backendStartTs=0, lastActivity=1677240741324, uri=/nl/res/xxx
/newsletters/xx/img/logofooter_telepass.png, queryString=, sslProtocol=TLSv1.3, cipherSuite=TLS_AES_128_GCM_SHA256, servedFromCache=false);
Error: io.netty.channel.ChannelException: io.netty.channel.unix.Errors$NativeIoException: newSocketStream(..) failed: Too many open files
eb 24, 2023 1:20:34 PM org.carapaceproxy.core.RequestsLogger rotateAccessLogFile
SEVERE: Error: Unable to rename file /data/carapace/accesslog/access.log in /data/carapace/accesslog/access.log-2023-02-24-34: java.nio.file.FileSystemException: /data/carapace/accesslog/access.log: Too many open files
Feb 24, 2023 1:20:34 PM org.carapaceproxy.core.RequestsLogger rotateAccessLogFile
SEVERE: Error: Unable to rename file /data/carapace/accesslog/access.log in /data/carapace/accesslog/access.log-2023-02-24-34: java.nio.file.FileSystemException: /data/carapace/accesslog/access.log: Too many open files
Feb 24, 2023 1:20:34 PM org.carapaceproxy.core.RequestsLogger rotateAccessLogFile
SEVERE: Error: Unable to rename file /data/carapace/accesslog/access.log in /data/carapace/accesslog/access.log-2023-02-24-34: java.nio.file.FileSystemException: /data/carapace/accesslog/access.log: Too many open files
Feb 24, 2023 1:20:34 PM org.carapaceproxy.core.RequestsLogger rotateAccessLogFile
SEVERE: Error: Unable to rename file /data/carapace/accesslog/access.log in /data/carapace/accesslog/access.log-2023-02-24-34: java.nio.file.FileSystemException: /data/carapace/accesslog/access.log: Too many open files
hamadodene commented 1 year ago

The problem seems to be related to too many open sockets. The excessive number of open sockets was due to a high number of requests that were arriving at our production shells.

In standard situations the number of open sockets is around 40k peak. While in this case we had reached a number of open sockets >65k.

image

hamadodene commented 1 year ago

Maybe #416 can help improve the situation. But I'm not sure. I will try to do some tests.