Closed zzack closed 10 years ago
502 bad gateway is a sign of problems with php-fpm, or rather with timeouts, and next mysql timeouts. To investigate what causes that, please monitor server logs:
$ tail -f /var/log/php/php-fpm-slow.log
$ tail -f /var/log/php/error_log
$ tail -f /var/log/syslog
Check also available memory and swap to find if you need to tune your php-fpm and mysql configuration, especially when you are running it on the server with less than 1 GB of RAM.
Thanks! If it happens again I will check those logs, and report back if I find anything that might be generally useful to people.
Please leave issues open so others can easier find it. We will have better issue tracker, but for now we are using github native system.
A visit to the drupal 7 beta's module page is resulting in a 502 and the following error in syslog:
mysqld: 101013 17:20:55 [Warning] Aborted connection 71 to db: 'd7b1facepalmin' user: 'd7b1facepalmin' host: '69.164.198.158' (Got an error reading communication packets)
I am running on a 768MB linode.
Restarting php-fpm allows the page to display once before failing again so perhaps I should be adjusting php-fpm? Any tips?
For completion here is the php-fpm-error.log:
Oct 13 17:44:22.624080 [WARNING] fpm_stdio_child_said(), line 167: child 20473 (pool default) said into stderr: "[Wed Oct 13 17:44:22 2010] [apc-error] Cannot redeclare class databasetasks in /var/aegir/platforms/drupal-7.0-beta1/modules/system/system.admin.inc on line 790."
Oct 13 17:44:22.630600 [NOTICE] fpm_got_signal(), line 48: received SIGCHLD
Oct 13 17:44:22.630706 [WARNING] fpm_children_bury(), line 215: child 20473 (pool default) exited with code 2 after 662.760568 seconds from start
Oct 13 17:44:22.632835 [NOTICE] fpm_children_make(), line 352: child 21543 (pool default) started
So removing APC from php.ini fixes the problem but I don't know where do go from here.
It is a known already, however new issue with beta1. Previous alpha versions worked without any problem. Well, only alpha7 didn't allow to delete the site, but alpha6 worked fine. Thanks for reporting it, I hope it will help to debug it for the next beta.
Sorry - should I be reporting this in an already established d.o issue? I looked there but couldn't find one - or is a barracuda bug?
Oh and I can't delete the site with drupal 7 beta 1 either.
Dropping database d7arighthandface
Drush command could not be completed.
Output from failed command : Fatal error: Call to undefined function cache_get() in /data/disk/octopus/platforms/drupal-7.0-beta1/includes/module.inc on line 622
An error occurred at function : drush_hosting_task
In addition to memory and swap use, you should also check for excessive Wait-IO. I have also encountered this problem on a vps that is on a host with a lot disk activity.
see also: http://github.com/omega8cc/nginx-for-drupal/issues#issue/84
Thanks for the tip - what do you consider excessive W-IO ? vmstat is definitely reporting some non-zero numbers but not constant:
root@lefthand:~# vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 161520 17564 358852 0 0 51 23 161 109 2 1 94 3
0 0 0 160892 17572 358852 0 0 0 2 931 677 2 0 98 0
0 0 0 161148 17572 358852 0 0 0 118 1014 735 1 0 98 0
0 0 0 161512 17580 358852 0 0 0 31 901 683 2 0 98 0
0 0 0 159288 17588 358856 0 0 1 138 1105 775 2 0 98 0
0 0 0 159404 17596 358856 0 0 0 3 874 683 1 0 98 0
0 0 0 159536 17620 358860 0 0 1 191 995 722 1 0 97 1
0 0 0 158660 17632 358864 0 0 0 9 962 700 2 1 98 0
0 0 0 159272 17652 358900 0 0 1 214 2014 1084 6 2 91 1
0 0 0 159264 17660 358904 0 0 0 6 957 692 2 0 98 0
0 0 0 155304 17700 359580 0 0 63 269 1733 1125 3 1 96 0
0 0 0 153560 17740 359728 0 0 62 23 1227 864 2 0 95 2
0 0 0 125660 18152 372512 0 0 531 488 1480 892 3 0 91 6
0 0 0 125924 18176 372512 0 0 0 42 983 705 2 0 98 0
0 0 0 125676 18200 372512 0 0 26 100 1051 742 2 0 98 0
0 0 0 123916 18284 372984 0 0 72 82 1600 919 3 1 95 1
0 0 0 123676 18320 373016 0 0 15 218 1267 900 2 0 98 0
0 0 0 123932 18336 373020 0 0 0 138 1123 784 2 0 98 0
1 0 0 123932 18376 373020 0 0 0 212 1146 807 2 0 92 6
0 0 0 124056 18392 373032 0 0 1 13 1148 778 2 1 96 2
0 0 0 107532 18448 373076 0 0 12 389 2293 1214 8 2 89 1
0 0 0 124024 18464 373080 0 0 1 49 1045 737 2 0 98 0
0 0 0 123552 18496 373092 0 0 1 298 1194 791 2 1 95 2
Is this likely something that could be tuned a bit or is it a sign that barracuda really can't run on 768MB mem ?
Your Wait-IO is consistently under 10 and it does not look like your using any swap. Barracuda runs fine on 512. Its the resource intensive drupal installs that may have a problem when server the server is under heavy disk load.
Try another install now and see if you can reproduce the issue with the above load.
You mean do a clean reinstall on the same VPS ?
apc.include_once_override = 0
. It is not a Barracuda bug, however, rather something introduced in beta, which breaks when used with this option. There is no need to disable APC.There should no need for a new install of upgrade of Baradcuda
Simply create another site. Sorry for the confusion, Continue to watch your vmstats, the case I pointed to above shows how this is related to problems with tasks that do not show completed and how to manually delete them.
The problem is most likely intermittent and you my need the vmstat results showing high Wait-IO to open a support request with your hosting provider.
Here is and example vmstat output for a vps with 4 gig of ram on an overloaded sever:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 34072 51556 321452 522660 0 0 1 15 6 9 1 0 98 1 0 1 34072 51424 321520 523472 0 0 0 1520 1153 1057 4 1 80 15 1 1 34072 50804 321560 523868 0 0 0 440 1691 325 10 1 83 6 0 1 34072 50060 321628 524408 0 0 0 1004 691 619 2 0 81 16 1 0 34072 49564 321684 524812 0 0 0 1420 956 895 3 1 78 18 0 2 34072 48200 321768 525248 0 0 0 1192 744 727 2 1 79 18 1 0 34072 47580 321840 525828 0 0 0 1100 808 695 3 1 80 17 0 1 34072 47704 321936 526456 0 0 0 1732 1689 1640 3 1 77 19 0 2 34072 47580 321952 526492 0 0 0 6808 297 311 0 0 76 24 1 2 34072 47332 321960 526884 0 0 0 9792 536 271 1 0 71 28 0 2 34072 47332 321980 526868 0 0 0 3756 335 263 0 0 74 26 0 5 34072 47456 322000 526876 0 0 0 2060 420 454 1 0 69 30 2 0 34072 46836 322060 527464 0 0 0 1452 1017 985 3 1 79 16 0 1 34072 44728 322148 528248 0 0 0 1552 870 767 3 1 80 15 2 0 34072 45348 322200 528672 0 0 0 992 650 624 2 0 77 20 0 1 34072 43744 322300 529472 0 0 0 1640 1063 980 4 1 81 14 2 0 34072 43868 322356 529896 0 0 0 1092 791 746 3 1 78 19 0 1 34072 43620 322404 530484 0 0 0 948 1320 1202 1 1 75 22 1 1 34072 20964 322452 530532 0 0 0 840 1578 2109 6 4 69 21 0 1 34072 19896 322488 530548 0 0 0 476 382 330 0 0 76 24
Thanks for the help - it doesn't look like swapping is the issue - php is just dying for some reason.
It's gone down again, no sites accessible and hosting-dispatch jobs piling up, eventually that will cause it to swap but at moment it's holding.
Back to the drawing board.
I think it got something to do with backlog check kernel parameters net.ipv4.tcp_max_syn_backlog and net.core.somaxconn
try to increase them and set value in php-fpm.conf for
I'm still not sure which values are enough/too high, I saw some documents which recommend to give it a value in the thousands (3000) but try to experiment with it.
after setting kernel parameter you probably need restart of the php-fpm and nginx
btw - if its already solved please state that it is solved, and how :)
Hi,
I'm running a Linode 512 VPS
I just ran an upgrade on Barracuda, as i was having trouble cloning, i'm now getting 502 Bad Gateway on all sites inc Aegir
I ran these commands as suggested
$ tail -f /var/log/php/php-fpm-slow.log
just sits there with 'rotate'
$ tail -f /var/log/php/error_log
[11-Apr-2011 09:57:38] PHP Fatal error: PHP Startup: apc_mmap: mmap failed: in Unknown on line 0
$ tail -f /var/log/syslog
Apr 11 10:00:53 aegir kernel: php-cgi[15777]: segfault at f ip 00007f919db9ac10 sp 00007fff43ce83a0 error 6 in libpthread-2.11.1.so[7f919db92000+18000] Apr 11 10:01:07 aegir kernel: php-cgi[15963]: segfault at f ip 00007f9080833c10 sp 00007fffbefb4f70 error 6 in libpthread-2.11.1.so[7f908082b000+18000]
Restarting the server gets it back on line, but it quickly dies again
Any idea's?
Current configuration is tuned for 1 GB of RAM minimum (see the README.txt). To use it with lower memory VPS please change the default APC value in the /opt/etc/php.ini
from apc.shm_size = 512M
to apc.shm_size = 128M
and restart it with service php-fpm restart
.
That seems to have done the trick - thanks
Ok, seems to be falling over eventually. On Linode graphs i see a big I/O spike and then the CPU maxes out and the server stops responding, I'll upgrade to 1 gig ram + i noticed a couple of errors in the console:
"Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start S20cron
start: Unknown job: S20cron
and also
Starting redis-server: failed "
This may or may not be a barracuda/octopus issue, but then again it may be. I have been running both for about a week with no problems. Randomly today I started receiving a 502 Bad Gateway error from both the barracuda and octopus sites, and all sites managed by the aegir installs.
Rebooting the server fixed the issue, but doing a google search for "aegir 502 bad gateway" shows a number of barracuda users are getting this error (although not with exactly the same symptoms).
Anyone else experiencing this issue?