ProxySQL crashes when certain data is synchronized when using ProxySQL Cluster.

otooi commented 3 years ago

[x] A clear description of the issue

I found that proxysql crashes when certain data is synchronized when using ProxySQL Cluster.

proxysql: pthread_mutex_lock.c:433: __pthread_mutex_lock_full: Assertion `INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust' failed.
Error: signal 6:
proxysql(_Z13crash_handleri+0x4c)[0xaaaaccdee5bc]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffb2b675b8]
/lib/aarch64-linux-gnu/atomics/libc.so.6(raise+0xb0)[0xffffb2603470]
 ---- proxysql(_Z13crash_handleri+0x4c) [0xaaaaccdee5bc] : crash_handler(int)

From what I've checked, it follows the case below.

Change admin variables -> crash
Change mysql variables -> crash
change mysql users -> sync
change mysql servers -> sync
change proxysql servers -> sync

I haven't tested any other cases. The four cases that I could clearly reproduce were: When I change admin variables, mysql variables and run load admin variables to runtime or load mysql variables to runtime, other nodes crashed. Nodes crashed over and over again. When I undo the changed values and execute load ~ to runtime, crash stopped.

[x] ProxySQL version ProxySQL version 2.2.0-80-g417dbbe, codename Truls
[x] OS version I am using proxysql on AWS EC2 Graviton2. So I compiled and used ProxySQL with ARM. Operating System: Ubuntu 18.04.5 LTS Kernel: Linux 5.4.0-1054-aws Architecture: arm64
[x] The steps to reproduce the issue

I have configured a ProxySQL cluster with 3 nodes. The same happens when I change the number of nodes. I initialized it with the /etc/proxysql.cnf file below. The admin-cluster_ ~ _diffs_before_sync option used the default of 3, but I changed it to 1 for testing. (Same issue in 3)

datadir="/proxysql"

admin_variables=
{
    amysql_ifaces="0.0.0.0:6032"
    admin_credentials="admin:admin;cluster:cluster"
    cluster_username="cluster"
    cluster_password="cluster"
    cluster_proxysql_servers_diffs_before_sync=1
    cluster_mysql_variables_diffs_before_sync=1
    cluster_admin_variables_diffs_before_sync=1
    cluster_ldap_variables_diffs_before_sync=1
}

mysql_variables=
{

    interfaces="0.0.0.0:3306;0.0.0.0:3307"
    default_schema="information_schema"
    server_version="5.7.12"
    connection_warming="true"
    monitor_writer_is_also_reader="false"
    free_connections_pct=50
    query_cache_size_MB=2048

    connect_retries_on_failure=10
    connect_timeout_client=20000
    connect_timeout_server=20000
    connect_timeout_server_max=20000

    ping_interval_server_msec=60000
    ping_timeout_server=5000
    monitor_ping_timeout=5000

    shun_on_failures=60
    shun_recovery_time_sec=1

    query_retries_on_failure=200
    poll_timeout_on_failure=1000
    log_unhealthy_connections="false"
}

After initializing ProxySQL, connect to proxysql admin of each node and register proxysql_server.

insert into proxysql_servers values
('10.0.91.46',6032,0,'server1'),
('10.0.55.60',6032,0,'server2'),
('10.0.67.32',6032,0,'server3);
LOAD PROXYSQL SERVERS TO RUNTIME; SAVE PROXYSQL SERVERS TO DISK;

After that, if I change the admin variables and mysql variables mentioned above, a crash occurs in nodes other than the node where SQL was executed.

[x] The full ProxySQL error log (default location: /var/lib/proxysql/proxysql.log) proxysql-cluster.log
- - If there is an older stable version of proxysql cluster, please let me know so I can use it until this bug is resolved.

Thank you

renecannao commented 3 years ago

Hi @otooi . Please try our packages. Specifically: https://github.com/sysown/proxysql/releases/download/v2.2.0/proxysql_2.2.0-ubuntu18_arm64.deb

At this stage it is not possible to determine if the issue you are facing is due to a compiler issue, a corrupted binary, some libraries, etc. If the issue persist, it is worth investigating further. Thanks

otooi commented 3 years ago

Hi @renecannao Thank you for reply.

I deleted(moved) the existing proxysql binary file and I installed the deb file from the link you sent. I did the same test, but the proxysql server crashed.

mv /usr/bin/proxysql /usr/local/proxysql_old;
dpkg -i proxysql_2.2.0-ubuntu18_arm64.deb

set mysql-long_query_time = 800;
load mysql variables to runtime;

I have attached the new log file. (Probably the same as before. And sensitive information is masked)

Thank you proxysql-cluster-2.log

renecannao commented 3 years ago

Thank you for trying our binaries. Do you have a core dump? That will be required to investigate it further.

otooi commented 3 years ago

This is the result of checking the library dependency of proxysql using ldd. Could you please check this also?

root@ip-10-0-55-60:~# ldd -v /usr/bin/proxysql
    linux-vdso.so.1 (0x0000ffffa0415000)
    libgnutls.so.30 => /usr/lib/aarch64-linux-gnu/libgnutls.so.30 (0x0000ffff9f5ca000)
    libpthread.so.0 => /lib/aarch64-linux-gnu/atomics/libpthread.so.0 (0x0000ffff9f59e000)
    libdl.so.2 => /lib/aarch64-linux-gnu/atomics/libdl.so.2 (0x0000ffff9f589000)
    libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff9f3f5000)
    libm.so.6 => /lib/aarch64-linux-gnu/atomics/libm.so.6 (0x0000ffff9f33d000)
    libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff9f319000)
    libc.so.6 => /lib/aarch64-linux-gnu/atomics/libc.so.6 (0x0000ffff9f1c1000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffffa03e9000)
    libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000ffff9f194000)
    libp11-kit.so.0 => /usr/lib/aarch64-linux-gnu/libp11-kit.so.0 (0x0000ffff9f082000)
    libidn2.so.0 => /usr/lib/aarch64-linux-gnu/libidn2.so.0 (0x0000ffff9f056000)
    libunistring.so.2 => /usr/lib/aarch64-linux-gnu/libunistring.so.2 (0x0000ffff9eed3000)
    libtasn1.so.6 => /usr/lib/aarch64-linux-gnu/libtasn1.so.6 (0x0000ffff9eeb2000)
    libnettle.so.6 => /usr/lib/aarch64-linux-gnu/libnettle.so.6 (0x0000ffff9ee6f000)
    libhogweed.so.4 => /usr/lib/aarch64-linux-gnu/libhogweed.so.4 (0x0000ffff9ee2d000)
    libgmp.so.10 => /usr/lib/aarch64-linux-gnu/libgmp.so.10 (0x0000ffff9edb0000)
    libffi.so.6 => /usr/lib/aarch64-linux-gnu/libffi.so.6 (0x0000ffff9ed98000)

    Version information:
    /usr/bin/proxysql:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libm.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libm.so.6
        libgcc_s.so.1 (GCC_3.3) => /lib/aarch64-linux-gnu/libgcc_s.so.1
        libgcc_s.so.1 (GCC_3.0) => /lib/aarch64-linux-gnu/libgcc_s.so.1
        libgcc_s.so.1 (GCC_4.2.0) => /lib/aarch64-linux-gnu/libgcc_s.so.1
        libdl.so.2 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libdl.so.2
        libstdc++.so.6 (GLIBCXX_3.4.14) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.19) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (CXXABI_1.3.1) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (CXXABI_1.3.8) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.18) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (CXXABI_1.3.5) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.22) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.9) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.15) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.11) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (CXXABI_1.3) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.20) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.21) => /usr/lib/aarch64-linux-gnu/libstdc++.so.6
        libpthread.so.0 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libpthread.so.0
        libc.so.6 (GLIBC_2.25) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libgnutls.so.30 (GNUTLS_3_4) => /usr/lib/aarch64-linux-gnu/libgnutls.so.30
    /usr/lib/aarch64-linux-gnu/libgnutls.so.30:
        libidn2.so.0 (IDN2_0.0.0) => /usr/lib/aarch64-linux-gnu/libidn2.so.0
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_2.25) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libc.so.6 (GLIBC_2.22) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libtasn1.so.6 (LIBTASN1_0_3) => /usr/lib/aarch64-linux-gnu/libtasn1.so.6
        libnettle.so.6 (NETTLE_6) => /usr/lib/aarch64-linux-gnu/libnettle.so.6
        libhogweed.so.4 (HOGWEED_4) => /usr/lib/aarch64-linux-gnu/libhogweed.so.4
    /lib/aarch64-linux-gnu/atomics/libpthread.so.0:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_PRIVATE) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /lib/aarch64-linux-gnu/atomics/libdl.so.2:
        ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_PRIVATE) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libstdc++.so.6:
        libm.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libm.so.6
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libgcc_s.so.1 (GCC_4.2.0) => /lib/aarch64-linux-gnu/libgcc_s.so.1
        libgcc_s.so.1 (GCC_3.3) => /lib/aarch64-linux-gnu/libgcc_s.so.1
        libgcc_s.so.1 (GCC_3.0) => /lib/aarch64-linux-gnu/libgcc_s.so.1
        libc.so.6 (GLIBC_2.18) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /lib/aarch64-linux-gnu/atomics/libm.so.6:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_PRIVATE) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /lib/aarch64-linux-gnu/libgcc_s.so.1:
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /lib/aarch64-linux-gnu/atomics/libc.so.6:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1
    /lib/aarch64-linux-gnu/libz.so.1:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libp11-kit.so.0:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libdl.so.2 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libdl.so.2
        libpthread.so.0 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libpthread.so.0
        libc.so.6 (GLIBC_2.26) => /lib/aarch64-linux-gnu/atomics/libc.so.6
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libidn2.so.0:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libunistring.so.2:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libtasn1.so.6:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libnettle.so.6:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libhogweed.so.4:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libnettle.so.6 (NETTLE_6) => /usr/lib/aarch64-linux-gnu/libnettle.so.6
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libgmp.so.10:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6
    /usr/lib/aarch64-linux-gnu/libffi.so.6:
        ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
        libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/atomics/libc.so.6

otooi commented 3 years ago

[proxysql_core.zip] remove link to security reason @renecannao I have attached the core dump file. Please check. Thank you.

sysown / proxysql

ProxySQL crashes when certain data is synchronized when using ProxySQL Cluster. #3565