search

pschlan / cron-job.org

cron-job.org Open Source project
GNU General Public License v2.0
1.47k stars 254 forks source link

Apache Close-Wait from cron-job.org #213

Closed honsberg closed 1 year ago

honsberg commented 1 year ago

Every Night at 0:00 my Apache Crashes. The Connections with "close-wait" increases massively and hits the max-connection and memory limit.

with my lack knowledge of apache and tcpip, i found out, that many of these connections come from cron-job.org.

root@Treasure-1:~# ss | grep 128.140.8.200
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61344
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61304
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13618
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13676
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13652
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47232
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37920
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47212
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13640
tcp   CLOSE-WAIT 25     0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14612
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13662
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14588
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47244
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47260
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13632
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61350
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61360
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47246
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37938
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37966
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47222
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14572
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37932
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14638
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47218
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13686
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47258
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37918
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61318
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47206
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37926
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14630
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14564
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37994
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61328
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13684
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61368
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37944
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14604
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13688
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37950
tcp   ESTAB      517    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:37980
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61346
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:47196
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14586
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:13704
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61382
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:14618
tcp   CLOSE-WAIT 518    0           [::ffff:192.168.50.101]:https      [::ffff:128.140.8.200]:61288

grafik

the only solution is currently to systemctl restart apache2 every morning.

any suggestions?

pschlan commented 1 year ago

CLOSE-WAIT usually indicates that we've closed the connection (e.g. due to a timeout) but Apache did not process/acknowledge the connection close yet by closing the socket on its side.

Do you see (some) of your cron jobs timing out?

It looks like that the request is somehow hanging on the server side. It's very hard to tell without knowing what exactly you're running on your server behind the cron job URL.

Feel free to send me the account name / cron job URL to info@cron-job.org, referencing this issue, and I can also check this from our side.