search

mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
9.03k stars 1.18k forks source link

502 bad gateway #2795

Closed mcmufffin closed 5 years ago

mcmufffin commented 5 years ago

Heyho! Leider bekomme ich aktuell einen 502 Bad Gateway error beim Aufruf vom Interface. Bin mir leider nicht sicher ob es mit irgendeinem Update (Docker auf Host oder Mailcow selbst) im zusammenhang steht.

In den Logs kann ich leider quasi überhaupt nichts finden und wäre für Input dankbar!

System information and quick debugging General logs:

Nginx: nginx-mailcow_1 | 2019/07/19 16:23:47 [error] 18#18: *22 connect() failed (113: Host is unreachable) while connecting to upstream, client: someclientip, server: somehost, request: "GET / HTTP/2.0", upstream: "fastcgi://172.22.1.10:9002", host: "somehost" nginx-mailcow_1 | 2019/07/19 16:24:05 [error] 18#18: *22 connect() failed (113: Host is unreachable) while connecting to upstream, client: someclientip, server: somehost, request: "GET / HTTP/2.0", upstream: "fastcgi://[fd4d:6169:6c63:6f77::e]:9002", host: "somehost"

PHPFPM: Uptime: 578 Threads: 45 Questions: 1176 Slow queries: 0 Opens: 44 Flush tables: 1 Open tables: 36 Queries per second avg: 2.034 MySQL @ 00b8d53114b4b2c9f77919b39dd4cb6f8588fd9a36c2ab8126855527a6f38684 MySQL is up-to-date Running DB init... Fixed _sogo_static_view Cleaned up memcached 1 1 1 1 1 1 1 1 1 1 1 1 1

Müsste der Output nicht evtl. detailierter sein?

Further information (where applicable):

andryyy commented 5 years ago

How about iptables?

Did you try to reboot the server?

Am 19.07.2019 um 16:33 schrieb mcmufffin notifications@github.com:

Heyho! Leider bekomme ich aktuell einen 502 Bad Gateway error beim Aufruf vom Interface. Bin mir leider nicht sicher ob es mit irgendeinem Update (Docker auf Host oder Mailcow selbst) im zusammenhang steht.

In den Logs kann ich leider quasi überhaupt nichts finden und wäre für Input dankbar!

System information and quick debugging General logs:

Nginx: nginx-mailcow_1 | 2019/07/19 16:23:47 [error] 18#18: 22 connect() failed (113: Host is unreachable) while connecting to upstream, client: someclientip, server: somehost, request: "GET / HTTP/2.0", upstream: "fastcgi://172.22.1.10:9002", host: "somehost" nginx-mailcow_1 | 2019/07/19 16:24:05 [error] 18#18: 22 connect() failed (113: Host is unreachable) while connecting to upstream, client: someclientip, server: somehost, request: "GET / HTTP/2.0", upstream: "fastcgi://[fd4d:6169:6c63:6f77::e]:9002", host: "somehost"

PHPFPM: Uptime: 578 Threads: 45 Questions: 1176 Slow queries: 0 Opens: 44 Flush tables: 1 Open tables: 36 Queries per second avg: 2.034 MySQL @ 00b8d53114b4b2c9f77919b39dd4cb6f8588fd9a36c2ab8126855527a6f38684 MySQL is up-to-date Running DB init... Fixed _sogo_static_view Cleaned up memcached 1 1 1 1 1 1 1 1 1 1 1 1 1

Müsste der Output nicht evtl. detailierter sein?

Further information (where applicable):

Your OS (is Apparmor or SELinux active?) CentOS 7, SElinux testweise aus Your virtualization technology (KVM/QEMU, Xen, VMware, VirtualBox etc.) KVM Your server/VM specifications (Memory, CPU Cores) 2VCore, 4GB Ram Don't try to run mailcow on a Synology or QNAP NAS, do you? Nope :D Docker and Docker Compose versions 18.09.8 Output of git diff origin/master, any other changes to the code? Nope — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

mcmufffin commented 5 years ago

Hi Andy!

the php-fpm container is in a constant reboot cycle, so i dont think its iptables? What output do you need? iptables -L -vn? Rebooting did unfortunately not help.

andryyy commented 5 years ago

Ah. Check mysql-mailcow then. Could be a broken SQL table.

Do you use watchdog? If yes, stop the container please and restart php-fpm-mailcow. Wait a longer time for the db init process to complete. Watchdog should not kill the container while it runs an upgrade, but you can try it anyway.

Am 19.07.2019 um 16:39 schrieb mcmufffin notifications@github.com:

Hi Andy!

the php-fpm container is in a constant reboot cycle, so i dont think its iptables? What output do you need? iptables -L -vn? Rebooting did unfortunately not help.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

mcmufffin commented 5 years ago

Thanks!

yes i use the watchdog container. I stopped it and restarted it (fpm & mysql) but cant see any difference. The FPM container is still stuck in a reboot cycle. Shouldnt the fpm give a little more output then just its uptime and the ammount of querries or is this the expected behaviour?

Sry im a little tired .. i allready checked mysql and it seems fine. Does not look like its stuck in a upgrade:

2019-07-19 16:35:35 139942743156032 [Note] mysqld (mysqld 10.2.25-MariaDB-1:10.2.25+maria~bionic) starting as process 1 ...
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Uses event mutexes
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Using Linux native AIO
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Number of pools: 1
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Using generic crc32 instructions
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Completed initialization of buffer pool
2019-07-19 16:35:35 139942018897664 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-07-19 16:35:35 139942743156032 [Note] InnoDB: Highest supported file format is Barracuda.
2019-07-19 16:35:36 139942743156032 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-07-19 16:35:36 139942743156032 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-07-19 16:35:36 139942743156032 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-07-19 16:35:36 139942743156032 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-07-19 16:35:36 139942743156032 [Note] InnoDB: Waiting for purge to start
2019-07-19 16:35:36 139942743156032 [Note] InnoDB: 5.7.26 started; log sequence number 58226338
2019-07-19 16:35:36 139941375899392 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2019-07-19 16:35:36 139942743156032 [Note] Server socket created on IP: '::'.
2019-07-19 16:35:36 139942743156032 [Warning] 'proxies_priv' entry '@% root@3ac9fc0d9913' ignored in --skip-name-resolve mode.
2019-07-19 16:35:36 139942106597120 [Note] Event Scheduler: scheduler thread started with id 6
2019-07-19 16:35:36 139942743156032 [Note] mysqld: ready for connections.
Version: '10.2.25-MariaDB-1:10.2.25+maria~bionic'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2019-07-19 16:35:37 139941375899392 [Note] InnoDB: Buffer pool(s) load completed at 190719 16:35:37
mcmufffin commented 5 years ago

Hi Andy,

i just tried to switch back to phpfpm:1.41 and this works!

melihsd commented 5 years ago

I still have this problem and get 502. I tried all versions out but my php-fpm will just not start to listen on port 9000,9001 and 9002 grafik and also the logs dont tell me anything: grafik

My sql is fine tho: grafik BUT MY IP6NAT WONT START grafik Here is my watchdog grafik

mcmufffin commented 5 years ago

Have you tried the latest 1.43?

melihsd commented 5 years ago

I did;

but to be fair; This is the second installation on the server; I dont have Problems with the first one; But i cant properly install this one for some reason. (Also ofc since i have multiple ips, i did assign alls ips different. Also i remove nginx port binds since i let the nginx container join my web network so my nginx installation on the web network will reverse proxy the container without problems. It does work everything but php will not start at all. )

I will remove the image tag and so it will build ht elocal entrypoint and will add echo checkpoints to see where it stucks

melihsd commented 5 years ago

UPDATE:

The PHP container stucks at this point:

CONTAINER_ID=
# Todo: Better check if upgrade failed
# This can happen due to a broken sogo_view
[ -s /mysql_upgrade_loop ] && SQL_LOOP_C=$(cat /mysql_upgrade_loop)
until [[ ! -z "${CONTAINER_ID}" ]] && [[ "${CONTAINER_ID}" =~ ^[[:alnum:]]*$ ]]; do
  echo "CHECK: container_id "
  CONTAINER_ID=$(curl --silent --insecure https://dockerapi/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" 2> /dev/null | jq -rc "select( .name | tostring | contains(\"mysql-mailcow\")) | .id" 2> /dev/null)
done

for some reason it tries forever to get the Container ID it jsut spams "CHECK: container_id"

my mysql container is named: grafik

andryyy commented 5 years ago

Two cows on a single server is problematic at the moment.

melihsd commented 5 years ago

I found the fix so php can start:

Php is trying to get the Container ID opf mysql, but this wont work since its gettting the container id by its name "mysq-mailcow" and there will be mutliple.
I did the trick by limiting the search onto the composer project name. There is already an existing pull request abut this.

andryyy commented 5 years ago

This is not the only problem when trying to run multiple cows on a single server. This is not complete.

Someone already provided a PR which was actually broken.

melihsd commented 5 years ago

yes there is a new one which is maybe working i will test it

André Peters notifications@github.com schrieb am So., 21. Juli 2019, 15:59:

This is not the only problem when trying to run multiple cows on a single server. This is not complete.

Someone already provided a PR which was actually broken.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mailcow/mailcow-dockerized/issues/2795?email_source=notifications&email_token=AC7FDV5MWLLD4VKAKHTMPVTQARTURA5CNFSM4IFGSEDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2OEDXY#issuecomment-513556959, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7FDV2GZMZZIG7N75XNSQTQARTURANCNFSM4IFGSEDA .

andryyy commented 5 years ago

See my comments in this PR. :)

Am 21.07.2019 um 16:08 schrieb Melih Demirtas notifications@github.com:

yes there is a new one which is maybe working i will test it

André Peters notifications@github.com schrieb am So., 21. Juli 2019, 15:59:

This is not the only problem when trying to run multiple cows on a single server. This is not complete.

Someone already provided a PR which was actually broken.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mailcow/mailcow-dockerized/issues/2795?email_source=notifications&email_token=AC7FDV5MWLLD4VKAKHTMPVTQARTURA5CNFSM4IFGSEDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2OEDXY#issuecomment-513556959, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7FDV2GZMZZIG7N75XNSQTQARTURANCNFSM4IFGSEDA .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

mkuron commented 5 years ago

See #2796