HttpListener Too many files open

franciscopaniskaseker commented 4 years ago

I rebooted my server due #196 problem to fix some services, but I saw before reboot the error:

HttpListener::acceptConnection(): Accept failed: Too many file open files

I ignored this error due #196 problem, but after reboot with less than 1 minute of uptime I am seeing a lot of new log about the same thing

I checked the file "/etc/security/limits.conf" and I saw:

# End of file
* hard nofile 1000000
* soft nofile 1000000
* hard nproc 1000000
* soft nproc 1000000

So there is a big limit.

In file "/etc/sysctl.conf" I saw:

fs.file-max = 1000000
fs.nr_open = 1000000

So something is very wrong with lsws.

Any idea how to troubleshoot this?

qtwrk commented 4 years ago

Please try increase both max connection and max SSL connection

franciscopaniskaseker commented 4 years ago

Please try increase both max connection and max SSL connection

from lsws service? where lsws can be configured?

qtwrk commented 4 years ago

in Webadmin console , in https://SERVER_IP:7080

port is defualt OFF in firewall , you need to open it , and set password with command

/usr/local/lsws/admin/misc/admpass.sh

franciscopaniskaseker commented 4 years ago

Thank you. I am seeing all configuratiobs about lsws now.

cagivacode commented 4 years ago

Finally someone else is seeing this issue.....I have resorted to nightly restarts of lsws because of this.

On centos systemd ignores limits.conf.

I narrowed it down to the number of cache files that OLS is creating from the wordpress lscache plugin on my server.

lsof -n -l +D '/usr/local/lsws/cachedata' | wc -l

will give you that number. If I run a linkchecker I can cause the issue in a few minutes. Since the plugin crawls to fill the cache with OLS it will cause the problem by itself over time.

I have not been able to verify a fix because the too many files issue actually causes a number of non LSWS issue as well (e.g. certificate errors on the mail server and some webmail issues (rainloop / roundcube).

@qtwrk has seen my issue before on the other board

hennaboy commented 4 years ago

Try this

https://topic.alibabacloud.com/a/centos-sets-ulimit-resource-limits-for-systemd-service_1_18_20033367.html

Set the global limit in /etc/systemd/system.conf

DefaultLimitCORE = infinity DefaultLimitNOFILE = 100000 DefaultLimitNPROC = 100000

franciscopaniskaseker commented 4 years ago

Today happened again. Openlitespeed team never saw this problem.

https://github.com/litespeedtech/openlitespeed/issues/188

franciscopaniskaseker commented 4 years ago

[root@mail fd]# lsof -n -l +D '/usr/local/lsws/cachedata' | wc -l
13
[root@mail fd]# 

[root@mail fd]# ls -l | wc -l
49999
[root@mail fd]# pwd
/proc/6366/fd
[root@mail fd]# cd /proc/6367/fd/
[root@mail fd]# ls -l | wc -l
50000
[root@mail fd]# 

[root@mail fd]# ls -l /proc/6367/fd/15766 
lrwx------. 1 nobody nobody 64 Sep  8 11:02 /proc/6367/fd/15766 -> socket:[757348056]
[root@mail fd]# ls -l /proc/6367/fd/1692
lrwx------. 1 nobody nobody 64 Sep  8 11:02 /proc/6367/fd/1692 -> socket:[741016970]
[root@mail fd]#

franciscopaniskaseker commented 4 years ago

Finally someone else is seeing this issue.....I have resorted to nightly restarts of lsws because of this.

We did that too. This server uptime is 42 days. The lshttpd is up to 2 weeks and 6 days. For some reason the restart stopped to worked. I will check why. But this only postpone the problem.

franciscopaniskaseker commented 4 years ago

Happened again and I described the problem on openlitespeed git https://github.com/litespeedtech/openlitespeed/issues/188 .

I am starting thinking if the decision to run as nobody user under cyberpanel environment is causing some kind of leak of file descriptors due wrong permissions in somewhere. @qtwrk What do you think?

franciscopaniskaseker commented 4 years ago

To make more clear the environment that is getting the problem:

max 150.000 file descritptor for any user
10.000 max connections and max ssl connections on openlitespeed
Connection Timeout 300s on openlitespeed
Keep Alive Timeout 5s on openlitespeed
the server have very low traffic. we are using low traffic server to test the cyberpanel/openlitespeed stabiltiy
centos 7 updated
cyberpanel / openlitespeed updated
we execute one time, everyday, the below command to try to avoid this problem. but it did not work.
```
systemctl restart lshttpd
```

franciscopaniskaseker commented 4 years ago

Is it normal a lot of file descriptors about lshttpd opened?

[root@mail lshttpd]# lsof  | egrep -i nobody | egrep -ic "/tmp/lshttpd/tameosw.sock"
257
[root@mail lshttpd]# lsof  | egrep -i nobody | egrep -ic "/tmp/lshttpd/apexuni.sock"
246

[root@mail lshttpd]# pwd
/tmp/lshttpd
[root@mail lshttpd]# ls -l | wc -l
124

meramsey commented 4 years ago

I recommend setting the ulimit globally up seen this many times.

Check like this: Nobody username="nobody" ;echo "Current ulimit: $(ulimit -n)"; echo ""; echo "Checking ${username}'s currently opened files count: "; sudo lsof| tr -s ' ' | cut -d ' ' -f3 | grep ${username} | wc -l ; echo ""; echo "Checking ${username}'s currently opened files sorted by command: "; sudo lsof -u ${username} |awk '{print $1}'|grep -v COMMAND|sort |uniq -c

Root username="root" ;echo "Current ulimit: $(ulimit -n)"; echo ""; echo "Checking ${username}'s currently opened files count: "; sudo lsof| tr -s ' ' | cut -d ' ' -f3 | grep ${username} | wc -l ; echo ""; echo "Checking ${username}'s currently opened files sorted by command: "; sudo lsof -u ${username} |awk '{print $1}'|grep -v COMMAND|sort |uniq -c

for user in $(awk -F: '{ print $1}' /etc/passwd); do echo -n "${user}:"; lsof -u $user | wc -l; done

Fix with this ulimit -n 131072

Super annoying issue alot of places and default centos has it set to 1024 which is too low.

Resources: https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:config:increasing-os-file-descriptor-limit https://www.basezap.com/guide-to-raise-ulimit-open-files-and-mysql-open-files-limit/ https://askubuntu.com/questions/181215/too-many-open-files-how-to-find-the-culprit https://forums.cpanel.net/threads/warning-fd-limit-is-lower-than-required-under-max-load.612523/ https://lists.freedesktop.org/archives/systemd-devel/2017-March/038496.html

franciscopaniskaseker commented 4 years ago

Fix with this ulimit -n 131072

My limit is 150.000. I have servers with 5 times more traffic that not use more than 100k of file descriptors. I know this depend about the site use, but I am sure this is wrong in this environment.

Code-Egg commented 4 years ago

Thank you. I am seeing all configuratiobs about lsws now.

So what is the current maximum connection for both HTTP and HTTPS?

franciscopaniskaseker commented 4 years ago

So what is the current maximum connection for both HTTP and HTTPS?

https://github.com/usmannasir/cyberpanel/issues/197#issuecomment-693941384

usmannasir / cyberpanel

HttpListener Too many files open #197