Aborted connection to db: 'teamspeak' user: 'root' (Got an error reading communication packets)

urbinek commented 4 years ago

After recent update my server is loosing connection to database. After service start it works for a while and then randomly start showing errors.

From client side i can't connect with "data base error" but if i try 4-5 it works eventually

docker-compose.yml

version: '3.1'

services:
  server:
    image: teamspeak
    networks:
      - overlay
    ports:
      - 9987:9987/udp
      - 10011:10011
      - 30033:30033
    volumes:
      - teamspeak:/var/ts3server
    environment:
      TS3SERVER_DB_PLUGIN: ts3db_mariadb
      TS3SERVER_DB_SQLCREATEPATH: create_mariadb
      TS3SERVER_DB_HOST: db
      TS3SERVER_DB_USER: root
      TS3SERVER_DB_PASSWORD: super-secret-password
      TS3SERVER_DB_NAME: teamspeak
      TS3SERVER_DB_WAITUNTILREADY: 30
      TS3SERVER_LICENSE: accept
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: any
    stdin_open: true
    tty: true

  db:
    image: mariadb
    networks:
      - overlay
    volumes:
      - teamspeak-db:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: super-secret-password
      MYSQL_DATABASE: teamspeak
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: any
    stdin_open: true
    tty: true

networks:
  overlay:

volumes:
  teamspeak:
    external: true
  teamspeak-db:
    external: true

db logs

[root@xen ~]# docker container logs teamspeak_db.1.vrp5og302g0kusg8ovqk2lh8s
2020-07-05 08:14:02+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.4+maria~focal started.
2020-07-05 08:14:02+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-07-05 08:14:02+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.4+maria~focal started.
2020-07-05  8:14:03 0 [Note] mysqld (mysqld 10.5.4-MariaDB-1:10.5.4+maria~focal) starting as process 1 ...
2020-07-05  8:14:03 0 [Note] InnoDB: Using Linux native AIO
2020-07-05  8:14:03 0 [Note] InnoDB: Uses event mutexes
2020-07-05  8:14:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-07-05  8:14:03 0 [Note] InnoDB: Number of pools: 1
2020-07-05  8:14:03 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
2020-07-05  8:14:03 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-07-05  8:14:03 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2020-07-05  8:14:03 0 [Note] InnoDB: Completed initialization of buffer pool
2020-07-05  8:14:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-07-05  8:14:03 0 [Note] InnoDB: 128 rollback segments are active.
2020-07-05  8:14:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-07-05  8:14:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-07-05  8:14:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-07-05  8:14:03 0 [Note] InnoDB: 10.5.4 started; log sequence number 180786167; transaction id 8166
2020-07-05  8:14:03 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-07-05  8:14:03 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-07-05  8:14:03 0 [Note] Server socket created on IP: '::'.
2020-07-05  8:14:03 0 [Warning] 'proxies_priv' entry '@% root@82f219c41d03' ignored in --skip-name-resolve mode.
2020-07-05  8:14:03 0 [Note] Reading of all Master_info entries succeeded
2020-07-05  8:14:03 0 [Note] Added new Master_info '' to hash table
2020-07-05  8:14:03 0 [Note] mysqld: ready for connections.
Version: '10.5.4-MariaDB-1:10.5.4+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2020-07-05  8:14:03 0 [Note] InnoDB: Buffer pool(s) load completed at 200705  8:14:03
2020-07-05 10:17:41 10 [Warning] Aborted connection 10 to db: 'teamspeak' user: 'root' host: '10.0.4.4' (Got an error reading communication packets)
2020-07-05 10:17:41 6 [Warning] Aborted connection 6 to db: 'teamspeak' user: 'root' host: '10.0.4.4' (Got an error reading communication packets)
2020-07-05 10:17:41 7 [Warning] Aborted connection 7 to db: 'teamspeak' user: 'root' host: '10.0.4.4' (Got an error reading communication packets)
2020-07-05 10:17:41 5 [Warning] Aborted connection 5 to db: 'teamspeak' user: 'root' host: '10.0.4.4' (Got an error reading communication packets)
2020-07-05 10:17:41 9 [Warning] Aborted connection 9 to db: 'teamspeak' user: 'root' host: '10.0.4.4' (Got an error reading communication packets)
2020-07-05 10:17:41 8 [Warning] Aborted connection 8 to db: 'teamspeak' user: 'root' host: '10.0.4.4' (Got an error reading communication packets)

server logs

[root@xen ~]# docker container logs teamspeak_server.1.4z6trkjgskm0cdxmhp9iywbyh
2020-07-05 08:14:01.643489|INFO    |ServerLibPriv |   |TeamSpeak 3 Server 3.12.1 (2020-03-27 10:38:47)
2020-07-05 08:14:01.643789|INFO    |ServerLibPriv |   |SystemInformation: Linux 4.9.75-29.el7.x86_64 #1 SMP Fri Jan 5 19:42:28 UTC 2018 x86_64 Binary: 64bit
2020-07-05 08:14:01.643823|INFO    |ServerLibPriv |   |Using hardware aes
2020-07-05 08:14:01.645915|INFO    |DatabaseQuery |   |dbPlugin name:    MariaDB plugin, version 3, (c)TeamSpeak Systems GmbH
2020-07-05 08:14:01.645979|INFO    |DatabaseQuery |   |dbPlugin version: 2
2020-07-05 08:14:03.571567|WARNING |Accounting    |   |Unable to open licensekey.dat, falling back to limited functionality
2020-07-05 08:14:03.584599|INFO    |Accounting    |   |Licensing Information
2020-07-05 08:14:03.584642|INFO    |Accounting    |   |licensed to       : Anonymous
2020-07-05 08:14:03.584673|INFO    |Accounting    |   |type              : No License
2020-07-05 08:14:03.584715|INFO    |Accounting    |   |starting date     : Sat Feb  1 00:00:00 2020
2020-07-05 08:14:03.584742|INFO    |Accounting    |   |ending date       : Mon Feb  1 00:00:00 2021
2020-07-05 08:14:03.584780|INFO    |Accounting    |   |max virtualservers: 1
2020-07-05 08:14:03.584805|INFO    |Accounting    |   |max slots         : 32
2020-07-05 08:14:04.041544|INFO    |              |   |Puzzle precompute time: 423
2020-07-05 08:14:04.045154|INFO    |FileManager   |   |listening on 0.0.0.0:30033, [::]:30033
2020-07-05 08:14:04.158772|INFO    |VirtualServerBase|1  |listening on 0.0.0.0:9987, [::]:9987
2020-07-05 08:14:04.159267|INFO    |Query         |   |listening for query on 0.0.0.0:10011, [::]:10011
2020-07-05 08:14:04.159430|INFO    |CIDRManager   |   |updated query_ip_whitelist ips: 127.0.0.1/32, ::1/128,
2020-07-05 08:14:26.606667|INFO    |VirtualServerBase|1  |client connected 'urbinek'(id:5) from 10.0.0.2:56022
2020-07-05 08:32:03.565281|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:35:03.565444|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:38:03.565083|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:41:03.565859|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:44:03.565612|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:47:03.564797|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:50:03.565556|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:53:03.564869|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:56:03.564829|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query
2020-07-05 08:59:03.565373|ERROR   |DatabaseQuery |   |db_open() select distinct c.client_id, c.server_id from (clients c left outer jo error: Lost connection to MySQL server during query

docker service inspect teamspeak_server teamspeak_db

[
    {
        "ID": "w9i92gpyl35o1e3h43kd5ve3x",
        "Version": {
            "Index": 2933
        },
        "CreatedAt": "2020-06-27T08:39:08.054165013Z",
        "UpdatedAt": "2020-07-04T19:19:06.908454795Z",
        "Spec": {
            "Name": "teamspeak_server",
            "Labels": {
                "com.docker.stack.image": "teamspeak",
                "com.docker.stack.namespace": "teamspeak"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "teamspeak:latest@sha256:12526f6e6f28dc8c05607c2bd9a739c7406c75a6a2ab677405a98ef7d63d5278",
                    "Labels": {
                        "com.docker.stack.namespace": "teamspeak"
                    },
                    "Env": [
                        "TS3SERVER_DB_HOST=db",
                        "TS3SERVER_DB_NAME=teamspeak",
                        "TS3SERVER_DB_PASSWORD=super-secret-password",
                        "TS3SERVER_DB_PLUGIN=ts3db_mariadb",
                        "TS3SERVER_DB_SQLCREATEPATH=create_mariadb",
                        "TS3SERVER_DB_USER=root",
                        "TS3SERVER_DB_WAITUNTILREADY=30",
                        "TS3SERVER_LICENSE=accept"
                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "TTY": true,
                    "OpenStdin": true,
                    "Mounts": [
                        {
                            "Type": "volume",
                            "Source": "teamspeak",
                            "Target": "/var/ts3server",
                            "VolumeOptions": {}
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {},
                    "Isolation": "default"
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "fdr9c0ubyv6hslyfw2eh5glfm",
                        "Aliases": [
                            "server"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "udp",
                        "TargetPort": 9987,
                        "PublishedPort": 9987,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 10011,
                        "PublishedPort": 10011,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 30033,
                        "PublishedPort": 30033,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "PreviousSpec": {
            "Name": "teamspeak_server",
            "Labels": {
                "com.docker.stack.image": "teamspeak",
                "com.docker.stack.namespace": "teamspeak"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "teamspeak:latest@sha256:12526f6e6f28dc8c05607c2bd9a739c7406c75a6a2ab677405a98ef7d63d5278",
                    "Labels": {
                        "com.docker.stack.namespace": "teamspeak"
                    },
                    "Env": [
                        "TS3SERVER_DB_HOST=db",
                        "TS3SERVER_DB_NAME=teamspeak",
                        "TS3SERVER_DB_PASSWORD=super-secret-password",
                        "TS3SERVER_DB_PLUGIN=ts3db_mariadb",
                        "TS3SERVER_DB_SQLCREATEPATH=create_mariadb",
                        "TS3SERVER_DB_USER=root",
                        "TS3SERVER_DB_WAITUNTILREADY=30",
                        "TS3SERVER_LICENSE=accept"
                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "TTY": true,
                    "OpenStdin": true,
                    "Mounts": [
                        {
                            "Type": "volume",
                            "Source": "teamspeak",
                            "Target": "/var/ts3server",
                            "VolumeOptions": {}
                        }
                    ],
                    "Isolation": "default"
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "fdr9c0ubyv6hslyfw2eh5glfm",
                        "Aliases": [
                            "server"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 0
                }
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "udp",
                        "TargetPort": 9987,
                        "PublishedPort": 9987,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 10011,
                        "PublishedPort": 10011,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 30033,
                        "PublishedPort": 30033,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "udp",
                        "TargetPort": 9987,
                        "PublishedPort": 9987,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 10011,
                        "PublishedPort": 10011,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 30033,
                        "PublishedPort": 30033,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "udp",
                    "TargetPort": 9987,
                    "PublishedPort": 9987,
                    "PublishMode": "ingress"
                },
                {
                    "Protocol": "tcp",
                    "TargetPort": 10011,
                    "PublishedPort": 10011,
                    "PublishMode": "ingress"
                },
                {
                    "Protocol": "tcp",
                    "TargetPort": 30033,
                    "PublishedPort": 30033,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "h1e8edhr8a98ydos95yv0te55",
                    "Addr": "10.0.0.86/24"
                },
                {
                    "NetworkID": "fdr9c0ubyv6hslyfw2eh5glfm",
                    "Addr": "10.0.4.19/24"
                }
            ]
        }
    },
    {
        "ID": "tw99yeiknc5mi0qixnf3uhemj",
        "Version": {
            "Index": 2933
        },
        "CreatedAt": "2020-06-27T08:39:05.851549377Z",
        "UpdatedAt": "2020-07-04T19:19:06.908436533Z",
        "Spec": {
            "Name": "teamspeak_db",
            "Labels": {
                "com.docker.stack.image": "mariadb",
                "com.docker.stack.namespace": "teamspeak"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "mariadb:latest@sha256:4d9bcef2c824eabaa7f46ccab616cb9311fc84d7fc0bb4b6ddd5d7d917851e40",
                    "Labels": {
                        "com.docker.stack.namespace": "teamspeak"
                    },
                    "Env": [
                        "MYSQL_DATABASE=teamspeak",
                        "MYSQL_ROOT_PASSWORD=9DCgyCtU88mvr5A5XjUZEWgphyoYpmTzPv0tOA"
                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "TTY": true,
                    "OpenStdin": true,
                    "Mounts": [
                        {
                            "Type": "volume",
                            "Source": "teamspeak-db",
                            "Target": "/var/lib/mysql",
                            "VolumeOptions": {}
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {},
                    "Isolation": "default"
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        },
                        {
                            "Architecture": "arm64",
                            "OS": "linux"
                        },
                        {
                            "Architecture": "ppc64le",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "fdr9c0ubyv6hslyfw2eh5glfm",
                        "Aliases": [
                            "db"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip"
            }
        },
        "PreviousSpec": {
            "Name": "teamspeak_db",
            "Labels": {
                "com.docker.stack.image": "mariadb",
                "com.docker.stack.namespace": "teamspeak"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "mariadb:latest@sha256:4d9bcef2c824eabaa7f46ccab616cb9311fc84d7fc0bb4b6ddd5d7d917851e40",
                    "Labels": {
                        "com.docker.stack.namespace": "teamspeak"
                    },
                    "Env": [
                        "MYSQL_DATABASE=teamspeak",
                        "MYSQL_ROOT_PASSWORD=9DCgyCtU88mvr5A5XjUZEWgphyoYpmTzPv0tOA"
                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "TTY": true,
                    "OpenStdin": true,
                    "Mounts": [
                        {
                            "Type": "volume",
                            "Source": "teamspeak-db",
                            "Target": "/var/lib/mysql",
                            "VolumeOptions": {}
                        }
                    ],
                    "Isolation": "default"
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        },
                        {
                            "Architecture": "arm64",
                            "OS": "linux"
                        },
                        {
                            "Architecture": "ppc64le",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "fdr9c0ubyv6hslyfw2eh5glfm",
                        "Aliases": [
                            "db"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 0
                }
            },
            "EndpointSpec": {
                "Mode": "vip"
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip"
            },
            "VirtualIPs": [
                {
                    "NetworkID": "fdr9c0ubyv6hslyfw2eh5glfm",
                    "Addr": "10.0.4.16/24"
                }
            ]
        }
    }
]

wjts commented 3 years ago

Have you found any solution or workaround? I am facing the exact same issue. Currently trying different db images, but its slow - problems only arise after some time.

urbinek commented 3 years ago

No.

I've increased timeouts in mariadb setttings:

# docker-compose/teamspeak/etc/mysql/mariadb.conf.d/70-server.cnf
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see

# this is only for the mysqld standalone daemon
[mysqld]

# Description: Time in seconds that the server waits for a connect packet before
# returning a 'Bad handshake'. Increasing may help if clients regularly encounter
# 'Lost connection to MySQL server at 'X', system error: error_number' type-errors.
connect_timeout = 360

# Time in seconds to wait on writing a block to a connection before aborting the write.
net_write_timeout = 360
net_read_timeout = 360

And errors from db container were gone, but ts still have same issues.

In my case, i've just removed mariadb and went with sqlite

wjts commented 3 years ago

Thanks for the tip. I've downgraded mariadb image to 10.4 (precisely MARIADB_VERSION=1:10.4.13+maria~focal) and its now working for 3 days straight - the only logs i've got so far are db timeouts, but these warnings do not affect the uptime. 2020-07-23 16:45:47 1752 [Warning] Aborted connection 1752 to db: 'teamspeak' user: 'root' host: '10.0.4.4' (Got timeout reading communication packets) Guess it's time to consider migrating over to sqlite too.

malobre commented 3 years ago

With mariadb:10 and teamspeak:3.13, I was able to stop the errors by setting TS3SERVER_DB_CONNECTIONS to 1 in the teamspeak container.

racoon-one commented 2 years ago

With mariadb:10 and teamspeak:3.13, I was able to stop the errors by setting TS3SERVER_DB_CONNECTIONS to 1 in the teamspeak container.

From https://hub.docker.com/_/teamspeak/ TS3SERVER_DB_CONNECTIONS variable controls how many concurrent connections to the database are being used. Must be at least 2 and at most 100. Defaults to 10.

I am confused how this fix even works. But apparently it does.

TeamSpeak-Systems / teamspeak-linux-docker-images

Aborted connection to db: 'teamspeak' user: 'root' (Got an error reading communication packets) #59