Plugin 'AUDIT' init function returned error on RHEL 8 and MySQL 8.0.25

[cichomitiko]

Hello all,

Red Hat Enterprise Linux release 8.4 (Ootpa) mysql-community-server-8.0.25-1.el8.x86_64

offset-extract.sh returns:

# bash offset-extract.sh /usr/sbin/mysqld
GDB failed!!!

The gdb.txt reports:

{"8.0.25","98d08989fa4730c3ef5036e5652ca873"offsets.gdb:7: Error in sourced command file:
No symbol table is loaded.  Use the "file" command.

We have the following settings:

audit_validate_checksum        = OFF
#audit_offsets                  = 8536, 8584, 4056, 5528, 520, 0, 0, 32, 64, 160, 600, 8700, 5160, 4208, 4216, 4220, 6832, 1616, 32, 7792, 7832, 7816, 11624, 140, 664, 320
plugin-load                    = AUDIT=libaudit_plugin.so
audit_force_record_logins      = 1
audit_json_log_file            = /app/mysql/log/mysql-audit.log
audit_json_file                = 1
audit_record_cmds              = 'Connect,Quit,Failed Login'

This is the error log:

2021-07-15T14:33:54.904143+02:00 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.25) starting as process 18604
2021-07-15T14:33:54.914284+02:00 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-07-15T14:33:55.125920+02:00 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-07-15T14:33:55.208827+02:00 0 [Warning] [MY-000080] [Server] option 'audit-json-file-bufsize': signed value 0 adjusted to 1.
2021-07-15T14:33:55.208941+02:00 0 [Warning] [MY-000080] [Server] option 'plugin-audit-json-file-bufsize': signed value 0 adjusted to 1.
2021-07-15T14:33:55.212715+02:00 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-07-15T14:33:55.320598+02:00 0 [ERROR] [MY-010202] [Server] Plugin 'AUDIT' init function returned error.
2021-07-15T14:33:55.320740+02:00 0 [ERROR] [MY-010734] [Server] Plugin 'AUDIT' registration as a AUDIT failed.
2021-07-15T14:33:55.375365+02:00 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-07-15T14:33:55.375549+02:00 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-07-15T14:33:55.401722+02:00 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.25'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.

I've tried installing mysql-community-debuginfo-8.0.25-1.el8.x86_64.rpm and mysql-community-server-debug-8.0.25-1.el8.x86_64.rpm:

# bash offset-extract.sh /usr/sbin/mysqld /usr/lib/debug/usr/sbin/mysqld-debug-8.0.25-1.el8.x86_64.debug
//offsets for: /usr/sbin/mysqld (8.0.25)
{"8.0.25","98d08989fa4730c3ef5036e5652ca873", 8544, 8584, 4064, 5536, 520, 0, 0, 32, 64, 160, 608, 8700, 5168, 4208, 4216, 4220, 6840, 1656, 32, 7800, 7840, 7824, 11632, 140, 664, 320},

After setting the offsets, I've got:

2021-07-15T14:43:43.451551+02:00 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.25) starting as process 19042
2021-07-15T14:43:43.461115+02:00 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-07-15T14:43:43.679438+02:00 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-07-15T14:43:43.774102+02:00 0 [Warning] [MY-000080] [Server] option 'audit-json-file-bufsize': signed value 0 adjusted to 1.
2021-07-15T14:43:43.774232+02:00 0 [Warning] [MY-000080] [Server] option 'plugin-audit-json-file-bufsize': signed value 0 adjusted to 1.
2021-07-15T14:43:43.778196+02:00 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-07-15T14:43:43.883201+02:00 0 [ERROR] [MY-000000] [Server] Audit Plugin: unable to disassemble at address: 0xdd0c40. Aborting.
2021-07-15T14:43:43.883347+02:00 0 [ERROR] [MY-000000] [Server] McAfee Audit Plugin: unable to hot patch mysql_execute_command (0xdd0c40). res: -1.
2021-07-15T14:43:43.883497+02:00 0 [ERROR] [MY-010202] [Server] Plugin 'AUDIT' init function returned error.
2021-07-15T14:43:43.883647+02:00 0 [ERROR] [MY-010734] [Server] Plugin 'AUDIT' registration as a AUDIT failed.
2021-07-15T14:43:43.948718+02:00 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-07-15T14:43:43.948923+02:00 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-07-15T14:43:43.980147+02:00 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.25'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.

Any help could be greatly appreciated!

Best regards Dimitre

P.S. Just wanted to add that I tried with MySQL 8.0.22, 23 and 24 and the initialization fails with the same error.

[dverbeeck]

@cichomitiko We were able to work around this by rebuilding the mysql-community packages. It is highly likely your mysqld was compiled w/ -fcf-protection enabled which hardens the code and also makes it impossible to hotpatch the necessary memory locations

On RHEL, we circumvented by modifying the flags included in the rpmrc file, not sure where these flags are set in Debian but I assume apt has similar wrappers for compiling packages. I would start your investigation there.

[cichomitiko]

@dverbeeck, thank you for the information and for working on the issue!