OpenVAS vulnerability scan causes ProxySQL 2.0.0-rc2 crash

[unki]

Hello!

I have the situation where a vulnerability scan with OpenVAS (latest version) takes down a ProxySQL instance - approx. in 33% of my attempts.

Side note: I've installed proxysql-rc2_2.0.0-debian9_amd64.deb on a Debian Stretch (kernel 4.9.0-8-amd64) installation, but proxysql --version still identifies itself as a rc1.

ProxySQL rev. v2.0.0-rc1-17-g832aa48 -- Fri Sep 21 12:59:27 2018
Copyright (C) 2013-2018 ProxySQL LLC
This program is free and without warranty

• I'm using ProxySQL with a MariaDB Galera Cluster as backend nodes.
• Additionally SSL/TLS is configured (for frontend & backend) - so it could be related by the OpenVAS SSL vulnerability scans.

What happens is that ProxySQL records a crash in proxysql.log and then seems to automatically restart - but claims being unable to bind to port 6032/tcp again:

Error: signal 11:
proxysql(_Z13crash_handleri+0x1a)[0x55df1a9a0b3a]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7ff6a60fc060]
2019-01-06 04:37:45 main.cpp:1297:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-01-06 04:37:45 [INFO] Angel process started ProxySQL process 26736
Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Sat Sep 29 20:31:06 2018
Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Tue Jun 26 15:14:16 2018
Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Tue Jun 26 15:14:16 2018
Standard ProxySQL Admin rev. 0.2.0902 -- ProxySQL_Admin.cpp -- Sat Sep 29 20:31:06 2018
Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Sat Sep 29 20:31:06 2018
Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Fri Sep 21 12:59:27 2018
.... (some mysql_* table dumps) ...
2019-01-06 04:37:45 [INFO] Loading Galera info for (1,3,2,4,on,1,1,0,"dbprod.example.com")
Standard Query Processor rev. 0.4.0926 -- Query_Processor.cpp -- Sat Sep 29 20:31:06 2018
In memory Standard Query Cache (SQC) rev. 1.2.0905 -- Query_Cache.cpp -- Fri Sep 21 12:59:27 2018
2019-01-06 04:37:45 network.cpp:53:listen_on_port(): [ERROR] bind(): Address already in use
Standard MySQL Monitor (StdMyMon) rev. 1.2.0723 -- MySQL_Monitor.cpp -- Fri Sep 21 12:59:27 2018
2019-01-06 04:37:47 MySQL_HostGroups_Manager.cpp:3991:update_galera_set_writer(): [WARNING] Galera: setting host db12.example.com:3306 as writer

The reason for not being able to bind on 6032/tcp seems to be leftover proxysql_galera_checker.sh processes after the ProxySQL restart

26667 ?        S      0:00 /bin/bash /usr/share/proxysql/tools/proxysql_galera_checker.sh 1 2 1 1 /var/lib/proxysql/proxysql_galera_checker.log
26758 ?        S      0:00 /bin/bash /usr/share/proxysql/tools/proxysql_galera_checker.sh 1 2 1 1 /var/lib/proxysql/proxysql_galera_checker.log

Those have actually invoked a MariaDB client process that is also hanging around after the ProxySQL restart:

26759 ?        S      0:00 mysql -uadmin -h localhost -P 6032 --protocol=tcp -Nse SELECT count(*) FROM mysql_servers WHERE hostgroup_id IN (2) AND status = 'ONLINE'

And the fun fact is, that this MariaDB client process is suddenly listening on port 6032 after the crash - and also on 33306/tcp which is the listen_port I'm using.

root@lb3 /var/lib/proxysql # lsof | grep -i 26759 | grep -i listen
mysql     26759                      root   25u     IPv4         1806402292       0t0        TCP *:33306 (LISTEN)
mysql     26759                      root   26u     IPv4         1806402293       0t0        TCP *:33306 (LISTEN)
mysql     26759                      root   27u     IPv4         1806402294       0t0        TCP *:33306 (LISTEN)
mysql     26759                      root   28u     IPv4         1806402295       0t0        TCP *:33306 (LISTEN)
mysql     26759                      root   35u     IPv4         1806408288       0t0        TCP *:6032 (LISTEN)

So it seems, in case of a crash, the proxysql_galera_checker.sh and with it the MariaDB client have somehow obtained the listening ports from the crashing ProxySQL instance.

So basically there are two issues:

• ProxySQL crashes by the OpenVAS scan
• in case of a crash, the automatic restart fails in case of using proxysql_galera_checker.sh via the scheduler, as the child processes of the crashing process somehow obtain the listening ports. And the newly starting ProxySQL instance is then unable to bind to the listing ports itself.

I know How-to-report-a-crash-bug - but do you need anything else beside what is mentioned there?

Regards, Andreas

[renecannao]

Hi Andreas,

Thank you for the report. The second issue is likely to be resolved already. I am not familiar with OpenVAS: can you help us with a quick step-by-step reproducible case?

Thanks

[unki]

Hi René,

You could do it with the virtual-appliance that OpenVAS is offering here.

• You can access the Greenbone Security Assistant (basically the WebUI of this OpenVAS suite) via the browser at https://$appliance_ip:9392.
• Then from the menu, select Scans -> Tasks
• Use the blue icon with the white star to create a new task
  • Give it a name
  • Create a new Scan Target by clicking on the blue icon with the white star here too
  • give the target a name
  • select Manual and enter the IP or DNS name of a ProxySQL server
  • everything else you can leave default, click on Create to close the New Target window
  • The newly created target should have been automatically selected as Scan Target, otherwise do it now manually
  • Set Alterable Task to yes
  • Set Scan Config to Full and fast
  • Leave everything else as it as, click on Create
• Now back in the Tasks overview page, scroll down to the newly created task (below the graphs, if they are already shown)
• In the row of the task, right in the action columns, you find the green icon with the white arrow to start that specific task.
• If you manually reload the page, you should see the scan being performed.
• And within 2-3min I get ProxySQL crashed :)

These are the ProxySQL settings that I've modified:

| admin-mysql_ifaces       | 0.0.0.0:6032;/tmp/proxysql_admin.sock     |
| admin-web_enabled        | true                                      |
| mysql-have_ssl           | true                                      |
| mysql-interfaces         | 0.0.0.0:33306;/tmp/proxysql.sock          |
| mysql-max_allowed_packet | 16777216                                  |
| mysql-max_connections    | 5000                                      |
| mysql-monitor_password   | xxxxxx          |
| mysql-monitor_username   | monitor                          |
| mysql-ssl_p2s_ca         | /etc/ssl/example.com/server_ca.pem  |
| mysql-ssl_p2s_cert       | /etc/ssl/lb3.example.com_crt.pem         |
| mysql-ssl_p2s_key        | /etc/ssl/private/lb3.example.com_key.pem |