wsrep-debug documentation missing or unclear

[janlindstrom]

We got a report about the lack of documentation on wsrep_debug, on both MariaDB knowledge base, and the Codership documentation.There is no information on:

• how the extra debug output looks like

• • It looks like normal error log format with INFO level like: 2024-09-06 14:26:19 2 [Note] WSREP: open: enter(4,none,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 2 [Note] WSREP: open: leave(4,idle,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 2 [Note] WSREP: before_command: enter(4,idle,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 4 [Note] WSREP: before_command: success(4,exec,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 4 [Note] WSREP: Cluster table is empty, not recovering transactions 2024-09-06 14:26:19 2 [Note] WSREP: after_command_before_result: enter(4,exec,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 2 [Note] WSREP: after_command_before_result: leave(4,result,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 2 [Note] WSREP: after_command_after_result_enter(4,result,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 2 [Note] WSREP: after_command_after_result: leave(4,idle,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 2 [Note] WSREP: close: enter(4,idle,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 2 [Note] WSREP: close: leave(4,quit,high priority,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 4 [Note] WSREP: cleanup: enter(4,quit,local,success,0,toi: -1,nbo: -1) 2024-09-06 14:26:19 4 [Note] WSREP: cleanup: leave(4,none,local,success,0,toi: -1,nbo: -1)

• what the SERVER, TRANSACTION, STREAMING, CLIENT [1] [2] flags do (I can somewhat guess for the first three, but have no idea whatsoever on CLIENT)

• • SERVER: MariaDB server code contains WSREP_DEBUG log writes, and these will be added to server error log -- TRANSACTION: Logging from wsrep-lib transaction is added to the error log -- STREAMING: Logging from streaming transactions in wsrep-lib is added to the error log -- CLIENT: Logging from wsrep-lib client state is added to the error log. -- All this information is useful only for issue troubleshooting and requires knowledge on implementation details.

• the KB entry on wsrep_debug [3] has some extra info on how DDL is logged on all nodes starting with 10.6 that is missing on the Enterprise Server page on wsrep_debug

• the related evs.debug_log_mask [4] setting in wsrep_galera_options is not mentioned on the wsrep_debug pages [1] [2] the actual enterprise server documentation page on evs.debug_log_mask [4] only says

• "Default: 0x1", is even more useless than the already mostly useless "Controls EVS debug logging. Only effective when wsrep_debug is on." [5] in the knowledge base

• Codership documentation [6] warns against having this enabled on production servers as it can expose passwords that are missing from both the knowledge base and enterprise server documentation -- This is not true for MariaDB. In wsrep_thd_query() where user query is exposed does not print all information when SQL_COMMAND is SET (e.g. SET PASSWORD) or SQLCOM_CREATE_USER (only CREATE USER is logged nothing else)

• the description text on the command line [1] and the system variable [2] detail pages differ even though it is the very same setting

• the KB entry on wsrep_debug [3] has some extra info on how DDL is logged on all nodes starting with 10.6 that is missing on the Enterprise Server page on wsrep_debug

2024-09-06 14:37:57 13 [Note] WSREP: TOI Begin: CREATE SEQUENCE seq start with 1 minvalue 1 maxvalue 1000000 increment by 0 cache 1000 nocycle EN GINE=InnoDB 2024-09-06 14:37:57 13 [Note] WSREP: enter_toi_local: enter(13,exec,local,success,0,toi: -1,nbo: -1) 2024-09-06 14:37:57 13 [Note] WSREP: poll_enter_toi: 3,0 2024-09-06 14:37:57 13 [Note] WSREP: enter_toi_local: leave(13,exec,toi,success,0,toi: 3,nbo: -1) 2024-09-06 14:37:57 13 [Note] WSREP: avoiding binlog rotate due to TO isolation: 1 2024-09-06 14:37:57 13 [Note] WSREP: TO END: 3: CREATE SEQUENCE seq start with 1 minvalue 1 maxvalue 1000000 increment by 0 cache 1000 nocycle EN GINE=InnoDB

• the related evs.debug_log_mask [4] setting in wsrep_galera_options is not mentioned on the wsrep_debug pages [1] [2]
• the actual enterprise server documentation page on evs.debug_log_mask [4] only says
• "Default: 0x1", is even more useless than the already mostly useless "Controls EVS debug logging. Only effective when wsrep_debug is on." [5] in the knowledge base
• Codership documentation [6] warns against having this enabled on production servers as it can expose passwords that are missing from both the knowledge base and enterprise server documentation
• This is not true for MariaDB. In wsrep_thd_query() where user query is exposed does not print all information when SQL_COMMAND is SET (e.g. SET PASSWORD) or SQLCOM_CREATE_USER (only CREATE USER is logged nothing else)
• the description text on the command line [1] and the system variable [2] detail pages differ even though it is the very same setting

[janlindstrom]

[1] https://mariadb.com/docs/server/ref/mdb/cli/mariadbd/wsrep-debug/ [2] https://mariadb.com/docs/server/ref/mdb/system-variables/wsrep_debug/ [3] https://mariadb.com/kb/en/galera-cluster-system-variables/#wsrep_debug [4] https://mariadb.com/docs/server/ref/mdb/wsrep_provider_options/evs.debug_log_mask/ [5] https://mariadb.com/kb/en/wsrep_provider_options/#evsdebug_log_mask [6] https://galeracluster.com/library/documentation/mysql-wsrep-options.html#wsrep-debug

[byte]

thank you for the comprehensive report @janlindstrom - @kirjaamo you can take the first stab at writing this, i will review when ready

[janlindstrom]

Here we need to remember there is differences between MariaDB wsrep patch and MySQL wsrep patch on what is in this wsrep-debug error logging.

[kirjaamo]

@byte My initial part is done. You can review.