Audit Plugin failed hot patch send_result_to_client in 5.6 and 5.7

[atze234]

Currently the audit Plugin isnt starting in mysql 5.6 and 5.7 community Edition from Oracle. It returns these Errors on Startup: 5.7:

2017-12-20T09:20:32.230384Z 0 [Note] InnoDB: 5.7.20 started; log sequence number 1388912
2017-12-20T09:20:32.230644Z 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql-data/ib_buffer_pool
2017-12-20T09:20:32.230839Z 0 [Note] Plugin 'FEDERATED' is disabled.
2017-12-20T09:20:32.231347Z 0 [Warning] option 'audit-json-file-bufsize': signed value 0 adjusted to 1
2017-12-20T09:20:32.231374Z 0 [Warning] option 'plugin-audit-json-file-bufsize': signed value 0 adjusted to 1
2017-12-20T09:20:32.233593Z 0 [Note] McAfee Audit Plugin: starting up. Version: 1.1.5 , Revision: 761 (64bit). MySQL AUDIT plugin interface version: 1025 (0x401). MySQL Server version: 5.7.20-log.
2017-12-20T09:20:32.233615Z 0 [Note] McAfee Audit Plugin: setup_offsets audit_offsets: (null) validate_checksum: 1 offsets_by_version: 1
2017-12-20T09:20:32.233888Z 0 [Note] InnoDB: Buffer pool(s) load completed at 171220 10:20:32
2017-12-20T09:20:32.304427Z 0 [Note] McAfee Audit Plugin: mysqld: /usr/sbin/mysqld (2766f7e403d4168141e414feced5dc95)
2017-12-20T09:20:32.304475Z 0 [Note] McAfee Audit Plugin: extended offsets validate res: MySQL thread id 123456, OS thread handle 0, query id 789 aud_tusr
2017-12-20T09:20:32.304481Z 0 [Note] McAfee Audit Plugin: Using offsets from offset version: 5.7.20 (1e793b9a2c327a27309b3ff8a6b5d731)
2017-12-20T09:20:32.304504Z 0 [Note] McAfee Audit Plugin: Set whitelist_cmds num: 3, value: BEGIN,COMMIT,PING
2017-12-20T09:20:32.304528Z 0 [Note] McAfee Audit Plugin: Set password_masking_cmds num: 8, value: CREATE_USER,GRANT,SET_OPTION,SLAVE_START,CREATE_SERVER,ALTER_SERVER,CHANGE_MASTER,UPDATE
2017-12-20T09:20:32.304598Z 0 [Note] McAfee Audit Plugin: Compile password_masking_regex  res: [1]
2017-12-20T09:20:32.304605Z 0 [Note] McAfee Audit Plugin: Set password_masking_regex  value: [identified(?:/\*.*?\*/|\s)*?by(?:/\*.*?\*/|\s)*?(?:password)?(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]|password(?:/\*.*?\*/|\s)*?\((?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"](?:/\*.*?\*/|\s)*?\)|password(?:/\*.*?\*/|\s)*?(?:for(?:/\*.*?\*/|\s)*?\S+?)?(?:/\*.*?\*/|\s)*?=(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]|password(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]]
2017-12-20T09:20:32.304613Z 0 [Note] McAfee Audit Plugin: Set json_socket_name str: [] value: [/var/run/db-audit/mysql.audit__data_mysql-data_3306]
2017-12-20T09:20:32.304641Z 0 [Note] McAfee Audit Plugin: bufsize for file [/var/log/mysql/mysql-audit.log]: 1. Value of json_file_bufsize: 1.
2017-12-20T09:20:32.304685Z 0 [Note] McAfee Audit Plugin: success opening file: /var/log/mysql/mysql-audit.log.
2017-12-20T09:20:32.304693Z 0 [Note] McAfee Audit Plugin: mem func addr: 0x7f874bb4e2c0 mem start addr: 0x7f874bb4f000 page size: 4096
2017-12-20T09:20:32.304698Z 0 [Note] Audit Plugin: hot patching function: 0x556b888b8da0, trampolineFunction: 0x7f874bb4f000 trampolinePage: 0x7f874bb4f000
2017-12-20T09:20:32.304789Z 0 [Note] McAfee Audit Plugin: hot patch for: mysql_execute_command (0x556b888b8da0) complete. Audit func: 0x7f874bb529b0, Trampoline address: 0x7f874bb4f000, size: 16, used: 30.
2017-12-20T09:20:32.304797Z 0 [Note] Audit Plugin: hot patching function: 0x556b8886eb60, trampolineFunction: 0x7f874bb4f020 trampolinePage: 0x7f874bb4f000
2017-12-20T09:20:32.304814Z 0 [ERROR] Audit Plugin: unable to disassemble at address: 0x0x556b8886eb67. Found relative addressing for instruction: [jnz 0x556b8886eb75]. Aborting.
2017-12-20T09:20:32.304821Z 0 [ERROR] McAfee Audit Plugin: unable to hot patch send_result_to_client (0x556b8886eb60). res: -1.
2017-12-20T09:20:32.304825Z 0 [ERROR] Plugin 'AUDIT' init function returned error.
2017-12-20T09:20:32.304830Z 0 [ERROR] Plugin 'AUDIT' registration as a AUDIT failed.
2017-12-20T09:20:32.304843Z 0 [Note] McAfee Audit Plugin: deinit
2017-12-20T09:20:32.304849Z 0 [Note] Audit Plugin: removing hot patching function: 0x556b888b8da0 targetPage: 0x556b888b8000 trampolineFunction: 0x7f874bb4f000

relevant output on 5.6:

2017-12-20 09:44:18 5634 [Note] InnoDB: 5.6.38 started; log sequence number 93329
2017-12-20 09:44:18 7f982f7fe700 InnoDB: Loading buffer pool(s) from /data/mysql-data/ib_buffer_pool
2017-12-20 09:44:18 5634 [Note] McAfee Audit Plugin: starting up. Version: 1.1.5 , Revision: 761 (64bit). MySQL AUDIT plugin interface version: 769 (0x301). MySQL Server version: 5.6.38-log.
2017-12-20 09:44:18 5634 [Note] McAfee Audit Plugin: setup_offsets audit_offsets: (null) validate_checksum: 1 offsets_by_version: 1
2017-12-20 09:44:18 7f982f7fe700 InnoDB: Buffer pool(s) load completed at 171220  9:44:18
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: mysqld: /usr/sbin/mysqld (24d597e8f9bcb4c60106760f119ea806)
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: extended offsets validate res: MySQL thread id 123456, OS thread handle 0x0, query id 789 aud_tusr
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: Using offsets from offset version: 5.6.38 (276cb0ab3e7c5b2ff265c97f1c9ccf10)
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: Set whitelist_cmds num: 3, value: BEGIN,COMMIT,PING
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: Set password_masking_cmds num: 8, value: CREATE_USER,GRANT,SET_OPTION,SLAVE_START,CREATE_SERVER,ALTER_SERVER,CHANGE_MASTER,UPDATE
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: Compile password_masking_regex  res: [1]
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: Set password_masking_regex  value: [identified(?:/\*.*?\*/|\s)*?by(?:/\*.*?\*/|\s)*?(?:password)?(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]|password(?:/\*.*?\*/|\s)*?\((?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"](?:/\*.*?\*/|\s)*?\)|password(?:/\*.*?\*/|\s)*?(?:for(?:/\*.*?\*/|\s)*?\S+?)?(?:/\*.*?\*/|\s)*?=(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]|password(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]]
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: Set json_socket_name str: [] value: [/var/run/db-audit/mysql.audit__data_mysql-data_3306]
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: bufsize for file [/var/log/mysql/mysql-audit.log]: 1. Value of json_file_bufsize: 1.
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: success opening file: /var/log/mysql/mysql-audit.log.
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: mem func addr: 0x7f985e75d010 mem start addr: 0x7f985e75e000 page size: 4096
2017-12-20 09:44:19 5634 [Note] Audit Plugin: hot patching function: 0x55e4abecce50, trampolineFunction: 0x7f985e75e000 trampolinePage: 0x7f985e75e000
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: hot patch for: mysql_execute_command (0x55e4abecce50) complete. Audit func: 0x7f985e760ed0, Trampoline address: 0x7f985e75e000, size: 16, used: 30.
2017-12-20 09:44:19 5634 [Note] Audit Plugin: hot patching function: 0x55e4abe91b80, trampolineFunction: 0x7f985e75e020 trampolinePage: 0x7f985e75e000
2017-12-20 09:44:19 5634 [ERROR] Audit Plugin: unable to disassemble at address: 0x0x55e4abe91b87. Found relative addressing for instruction: [jnz 0x55e4abe91b95]. Aborting.
2017-12-20 09:44:19 5634 [ERROR] McAfee Audit Plugin: unable to hot patch send_result_to_client (0x55e4abe91b80). res: -1.
2017-12-20 09:44:19 5634 [ERROR] Plugin 'AUDIT' init function returned error.
2017-12-20 09:44:19 5634 [ERROR] Plugin 'AUDIT' registration as a AUDIT failed.
2017-12-20 09:44:19 5634 [Note] McAfee Audit Plugin: deinit

My.cnf for audit plugin:

plugin-load             = AUDIT=libaudit_plugin.so
audit_json_file         = ON
audit_json_log_file     = /var/log/mysql/mysql-audit.log

Whats wrong there? Thanks for your help Jörg

[aharonrobbins]

Hi. Please try the latest version of the plugin. It has offsets for the versions you're using. Thanks!

[kubo]

@aharonrobbins I think you misunderstood the issue. @atze234 uses the latest or the previous of the latest version of the plugin. Offsets for 5.7.20 are in the plugin as follows.

2017-12-20T09:20:32.304481Z 0 [Note] McAfee Audit Plugin: Using offsets from offset version: 5.7.20 (1e793b9a2c327a27309b3ff8a6b5d731)

[aharonrobbins]

@atze234 Please provide more details on MySQL version, which Linux dist, where you got the MySQL from and so on. We test on the distributions from MySQL. Thanks.

[aharonrobbins]

I think I just had the 'Aha'. If you have selinux enabled, you will need to disable it for the plugin to work. Please check into that. Thanks.

[atze234]

Hi @aharonrobbins, we don't have SELinux or Apparmor enabled. Linux Dist: latest Debian Stretch 9.3 Mysql Dist: from Repositories at repo.mysql.com -> mysqld Ver 5.7.20 for Linux on x86_64 (MySQL Community Server (GPL)) Audit Plugin: Latest Dev Snapshot: 1.1.5-761 Thank you

[aharonrobbins]

Dang! I didn't see this comment until just now. I'm not sure why this is still failing for you. We will push a new dev snapshot shortly.

[aharonrobbins]

We pushed a new dev-snapshot and binary release last week. Had this made a difference for you? Thanks.

[atze234]

Hi, it isnt working with the latest dev snapshot :( Still this error message in mysql 5.7:

2018-01-29T11:02:22.645671Z 0 [Note] McAfee Audit Plugin: starting up. Version: 1.1.6 , Revision: 779 (64bit). MySQL AUDIT plugin interface version: 1025 (0x401). MySQL Server version: 5.7.20-log.
2018-01-29T11:02:22.645689Z 0 [Note] McAfee Audit Plugin: setup_offsets audit_offsets: (null) validate_checksum: 1 offsets_by_version: 1
2018-01-29T11:02:22.650437Z 0 [Note] InnoDB: Buffer pool(s) load completed at 180129 12:02:22
2018-01-29T11:02:22.720230Z 0 [Note] McAfee Audit Plugin: mysqld: /usr/sbin/mysqld (2766f7e403d4168141e414feced5dc95)
2018-01-29T11:02:22.720288Z 0 [Note] McAfee Audit Plugin: extended offsets validate res: MySQL thread id 123456, OS thread handle 0, query id 789 aud_tusr
2018-01-29T11:02:22.720297Z 0 [Note] McAfee Audit Plugin: Using offsets from offset version: 5.7.20 (1e793b9a2c327a27309b3ff8a6b5d731)
2018-01-29T11:02:22.720330Z 0 [Note] McAfee Audit Plugin: Set whitelist_cmds num: 3, value: BEGIN,COMMIT,PING
2018-01-29T11:02:22.720347Z 0 [Note] McAfee Audit Plugin: Set whitelist_users num: 1, value: xxxx
2018-01-29T11:02:22.720365Z 0 [Note] McAfee Audit Plugin: Set password_masking_cmds num: 8, value: CREATE_USER,GRANT,SET_OPTION,SLAVE_START,CREATE_SERVER,ALTER_SERVER,CHANGE_MASTER,UPDATE
2018-01-29T11:02:22.720439Z 0 [Note] McAfee Audit Plugin: Compile password_masking_regex  res: [1]
2018-01-29T11:02:22.720448Z 0 [Note] McAfee Audit Plugin: Set password_masking_regex  value: [identified(?:/\*.*?\*/|\s)*?by(?:/\*.*?\*/|\s)*?(?:password)?(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]|password(?:/\*.*?\*/|\s)*?\((?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"](?:/\*.*?\*/|\s)*?\)|password(?:/\*.*?\*/|\s)*?(?:for(?:/\*.*?\*/|\s)*?\S+?)?(?:/\*.*?\*/|\s)*?=(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]|password(?:/\*.*?\*/|\s)*?['|"](?<psw>.*?)(?<!\\)['|"]]
2018-01-29T11:02:22.720461Z 0 [Note] McAfee Audit Plugin: Set json_socket_name str: [] value: [/var/run/db-audit/mysql.audit__data_mysql-data_3306]
2018-01-29T11:02:22.720548Z 0 [Note] McAfee Audit Plugin: bufsize for file [/data/mysql-log/mysql-audit.log]: 1. Value of json_file_bufsize: 1.
2018-01-29T11:02:22.720615Z 0 [Note] McAfee Audit Plugin: success opening file: /data/mysql-log/mysql-audit.log.
2018-01-29T11:02:22.720627Z 0 [Note] McAfee Audit Plugin: mem func addr: 0x7f841f00d330 mem start addr: 0x7f841f00e000 page size: 4096
2018-01-29T11:02:22.720635Z 0 [Note] Audit Plugin: hot patching function: 0x55a09cfa0da0, trampolineFunction: 0x7f841f00e000 trampolinePage: 0x7f841f00e000
2018-01-29T11:02:22.720721Z 0 [Note] McAfee Audit Plugin: hot patch for: mysql_execute_command (0x55a09cfa0da0) complete. Audit func: 0x7f841f011a20, Trampoline address: 0x7f841f00e000, size: 16, used: 30.
2018-01-29T11:02:22.720732Z 0 [Note] Audit Plugin: hot patching function: 0x55a09cf56b60, trampolineFunction: 0x7f841f00e020 trampolinePage: 0x7f841f00e000
2018-01-29T11:02:22.720750Z 0 [ERROR] Audit Plugin: unable to disassemble at address: 0x0x55a09cf56b67. Found relative addressing for instruction: [jnz 0x55a09cf56b75]. Aborting.
2018-01-29T11:02:22.720768Z 0 [ERROR] McAfee Audit Plugin: unable to hot patch send_result_to_client (0x55a09cf56b60). res: -1.
2018-01-29T11:02:22.720776Z 0 [ERROR] Plugin 'AUDIT' init function returned error.
2018-01-29T11:02:22.720783Z 0 [ERROR] Plugin 'AUDIT' registration as a AUDIT failed.
2018-01-29T11:02:22.720799Z 0 [Note] McAfee Audit Plugin: deinit
2018-01-29T11:02:22.720807Z 0 [Note] Audit Plugin: removing hot patching function: 0x55a09cfa0da0 targetPage: 0x55a09cfa0000 trampolineFunction: 0x7f841f00e000

[aharonrobbins]

Thanks for the update. I don't have an immediate answer for you, and I don't know if or when I will be able to try to reproduce the issue.

I would suggest compiling MySQL from source (it's pretty easy to do) and seeing if the plugin will load into a locally compiled mysqld. You should first extract the offsets from what you compile locally and verify that they match what's in the plugin source code. If they differ, you can either add the offsets to the plugin and compile it, or put them into the /etc/my.cnf file.

Hope this helps.

[kubo]

2017-12-20T09:20:32.304797Z 0 [Note] Audit Plugin: hot patching function: 0x556b8886eb60, trampolineFunction: 0x7f874bb4f020 trampolinePage: 0x7f874bb4f000
2017-12-20T09:20:32.304814Z 0 [ERROR] Audit Plugin: unable to disassemble at address: 0x0x556b8886eb67. Found relative addressing for instruction: [jnz 0x556b8886eb75]. Aborting.

How about allocating trampolinePage near hot patching function by using mmap with a non-null first argument? When the distance is less than 2G, the hot patching succeeds.

Edited: The first argument of mmap should be an unused address in /proc/self/maps.

[aharonrobbins]

Can you set up a pull request to show how to do this? @atze234 will you be able to test it? Unfortunately, I don't have the cycles at the moment to work on this. Thanks!

[kubo]

Well, I have the skill to do it but I cannot do it. Sorry.

[kubo]

When MySQL is compiled from source, this issue will disappear, I guess.

MySQL executables distributed by Oracle are compiled as PIE. On the other hand MySQL compiled from source isn't PIE. When mysqld isn't PIE, it is mapped at 0x4000000 in the address space and trampolinePage is allocated under 2G by using mmap with MAP_32BIT flag.

[kubo]

The following is the reason of my suggestion.

I thought that it wasn't due to SELinux because hot patching for mysql_execute_command didn't fail. If it was due to SELinux, it failed.

2017-12-20T09:20:32.304797Z 0 [Note] Audit Plugin: hot patching function: 0x556b8886eb60, trampolineFunction: 0x7f874bb4f020 trampolinePage: 0x7f874bb4f000
2017-12-20T09:20:32.304814Z 0 [ERROR] Audit Plugin: unable to disassemble at address: 0x0x556b8886eb67. Found relative addressing for instruction: [jnz 0x556b8886eb75]. Aborting.

The hot patching failed due to relative addressing at address 0x556b8886eb67, which is 7th byte after the hot patching function at 0x556b8886eb60. When the distance from the hot patching function to trampolineFunction is less than 2G, 5 bytes are required to insert a jump instruction. On the other hand, when it is more than 2G, 14 bytes are required however relative addressing was found at the 7th byte. If the distance becomes less than 2G, the hot patching succeeds because there is no relative addressing within the first 6 bytes of the hot patching function.

[aharonrobbins]

Thanks for the explanation and links.

As I said earlier, I am not in a position right now to try your suggestion of a non-null first argument to mmap. I see now that you are also unable to do it. So perhaps @atze234 can try it and let us know...

Thanks.

[atze234]

Hi, sorry, unfortunately i dont have the C skills and knowledge of this addressing to change things here :(

[atze234]

Why this issue is closed? Is it fixed in a new version? Or should Oracles Mysql Distribution not be supported anymore? Maybe this can be fixed in the future?

[aharonrobbins]

Reopening. Unfortunately I don't have the cycles to work on this right now. I will leave it open for the future.

[egegunes]

@kubo was right on PIE executables:

Hot patching send_result_client succeeds with this build:

/usr/local/mysql/bin/mysqld: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=654322e6d4af29555f1e555b57d46004ac61f51d, not stripped

Hot patching fails on this:

/usr/local/mysql/bin/mysqld: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64. so.2, for GNU/Linux 2.6.32, BuildID[sha1]=e183498ec80d2d706b51507f076c32cef5286147, not stripped 

@aharonrobbins any intention to fix this issue? I'm willing to work on it but I'm going to need a significant assistance.

[aharonrobbins]

@egegunes Hello. I don't check this email account very often; I just now saw your comment.

I have not been affiliated with McAfee for close to 3 years. They moved development of their database security product, and with it the MySQL and Postgres plugins, to another site.

It is (sadly) rather clear that they've abandoned development of the plugins, but that's not anything I can influence. I also am not in a position to provide any assistance for work you may care to do on the plugin.

Sorry. Best wishes -- Aharon