search

percona / percona-xtradb-cluster-operator

Percona Operator for MySQL based on Percona XtraDB Cluster
https://www.percona.com/doc/kubernetes-operator-for-pxc/index.html
Apache License 2.0
542 stars 188 forks source link

Post initialization spec.pxc.configuration #1628

Open MAXxATTAXx opened 9 months ago

MAXxATTAXx commented 9 months ago

Proposal

Currently there is only one configuration file that is used for both mysql datadir initialization as well as mysql startup. It will be convenient to have a way to inject mysql configuration post data dir initialization as some plugins are not available when using "mysql --initialize" causing the datadir to not be initialized.

Use-Case

When trying to configure a cluster to write audit_logs by using percona audit_log plugin the cluster fails to initialize because of unknown variable errors if the audit_log configuration is specified. Currently there is no way to launch a cluster with a complete configuration of this plugin.

This causes a lot of headaches when trying to following GitOps principles as new Databases need to be deployed in a two step approach. First, deploy the cluster without audit_log configuration to allow it to initialize. Second, you need to update your configuration and add the missing audit_log variables, restart the cluster and have those configured.

By allowing a hypothetical spec.pxc.configuration_post_init configuration of plugins could be placed there and only loaded by mysql only after post-init. This would allow for a single configuration to bot hinitialize and manage clusters.

Is this a feature you are interested in implementing yourself?

Maybe

Anything else?

Use the following snippet if you want to test out the error:

spec:
  pxc:
    configuration: |
      [mysqld]
      # Enable audit_log
      audit_log_file=audit.log
      audit_log_strategy=PERFORMANCE
      audit_log_format=JSON
      audit_log_policy=LOGINS
      audit_log_exclude_accounts=xtrabackup@%,replication@%,proxyadmin@%,operator@%,monitor@%,clustercheck@%

Make sure you are launching on a uninitialized PVC. The following errors will show in /var/lib/mysql/mysqld-error.log:

2024-02-13T20:31:52.517092Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library 'none'
2024-02-13T20:31:52.523257Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-02-13T20:31:53.596678Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-02-13T20:31:53.596678Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-02-13T20:31:54.110877Z 0 [Warning] [MY-013501] [Server] Ignoring --plugin-load[_add] list as the server is running with --initialize(-insecure).
2024-02-13T20:31:54.704477Z 0 [ERROR] [MY-000067] [Server] unknown variable 'audit_log_file=audit.log'.
2024-02-13T20:31:54.704564Z 0 [ERROR] [MY-013455] [Server] The newly created data directory /var/lib/mysql//sst-xb-tmpdir/ by --initialize is unusable. You can remove it.
2024-02-13T20:31:54.705448Z 0 [ERROR] [MY-010119] [Server] Aborting
spron-in commented 8 months ago

Hello @MAXxATTAXx - this looks interesting. Do you think that it is related to #1620 ? Bootstrapping and initialization.

lanss315425 commented 7 months ago

I have the same question

Defaulted container "pxc" out of: pxc, pxc-init (init)
+ trap exit SIGTERM
+ '[' m = - ']'
+ CFG=/etc/mysql/node.cnf
+ wantHelp=
+ for arg in "$@"
+ case "$arg" in
++ mysqld -V
++ awk '{print $3}'
++ awk -F. '{print $1"."$2}'
+ MYSQL_VERSION=8.0
++ mysqld -V
++ awk '{print $3}'
++ awk -F. '{print $3}'
++ awk -F- '{print $1}'
+ MYSQL_PATCH_VERSION=35
+ vault_secret=/etc/mysql/vault-keyring-secret/keyring_vault.conf
+ '[' -f /etc/mysql/vault-keyring-secret/keyring_vault.conf ']'
+ '[' -f /usr/lib64/mysql/plugin/binlog_utils_udf.so ']'
+ sed -i '/\[mysqld\]/a plugin_load="binlog_utils_udf=binlog_utils_udf.so"' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a gtid-mode=ON' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a enforce-gtid-consistency' /etc/mysql/node.cnf
+ grep -q '^progress=' /etc/mysql/node.cnf
+ sed -i 's|^progress=.*|progress=1|' /etc/mysql/node.cnf
+ grep -q '^\[sst\]' /etc/mysql/node.cnf
+ grep -q '^cpat=' /etc/mysql/node.cnf
+ sed '/^\[sst\]/a cpat=.*\\.pem$\\|.*init\\.ok$\\|.*galera\\.cache$\\|.*wsrep_recovery_verbose\\.log$\\|.*readiness-check\\.sh$\\|.*liveness-check\\.sh$\\|.*get-pxc-state$\\|.*sst_in_progress$\\|.*sleep-forever$\\|.*pmm-prerun\\.sh$\\|.*sst-xb-tmpdir$\\|.*\\.sst$\\|.*gvwstate\\.dat$\\|.*grastate\\.dat$\\|.*\\.err$\\|.*\\.log$\\|.*RPM_UPGRADE_MARKER$\\|.*RPM_UPGRADE_HISTORY$\\|.*pxc-entrypoint\\.sh$\\|.*unsafe-bootstrap\\.sh$\\|.*pxc-configure-pxc\\.sh\\|.*peer-list$\\|.*auth_plugin$\\|.*version_info$' /etc/mysql/node.cnf
+ [[ 8.0 == \8\.\0 ]]
+ [[ 35 -ge 26 ]]
+ grep -q '^skip_replica_start=ON' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a skip_replica_start=ON' /etc/mysql/node.cnf
+ auth_plugin=caching_sha2_password
+ [[ -f /var/lib/mysql/auth_plugin ]]
++ cat /var/lib/mysql/auth_plugin
+ prev_auth_plugin=caching_sha2_password
+ [[ caching_sha2_password != \m\y\s\q\l\_\n\a\t\i\v\e\_\p\a\s\s\w\o\r\d ]]
+ [[ caching_sha2_password == \m\y\s\q\l\_\n\a\t\i\v\e\_\p\a\s\s\w\o\r\d ]]
+ [[ -z caching_sha2_password ]]
+ [[ 8.0 == \5\.\7 ]]
+ echo caching_sha2_password
+ sed -i /default_authentication_plugin/d /etc/mysql/node.cnf
+ [[ 8.0 == \8\.\0 ]]
+ [[ 35 -ge 27 ]]
+ sed -i '/\[mysqld\]/a authentication_policy=caching_sha2_password,,' /etc/mysql/node.cnf
+ file_env XTRABACKUP_PASSWORD xtrabackup xtrabackup
Percona XtraDB Cluster: Finding peers
2024/03/28 01:02:43 Peer finder enter
2024/03/28 01:02:43 Determined Domain to be default.svc.cluster.local
2024/03/28 01:02:43 Peer list updated
was []
now [10-233-83-116.cluster1-pxc-unready.pxc.svc.cluster.local]
2024/03/28 01:02:43 execing: /var/lib/mysql/pxc-configure-pxc.sh with stdin: 10-233-83-116.cluster1-pxc-unready.pxc.svc.cluster.local
2024/03/28 01:02:43 ++ hostname -I
++ awk ' { print $1 } '
+ NODE_IP=10.233.83.116
++ hostname -f
++ cut -d. -f2
+ CLUSTER_NAME=cluster1-pxc
+ SERVER_NUM=0
+ SERVER_ID=26270930
++ hostname -f
+ NODE_NAME=cluster1-pxc-0.cluster1-pxc.pxc.svc.cluster.local
+ NODE_PORT=3306
+ read -ra LINE
+ echo 'read line 10-233-83-116.cluster1-pxc-unready.pxc.svc.cluster.local'
read line 10-233-83-116.cluster1-pxc-unready.pxc.svc.cluster.local
++ getent hosts 10-233-83-116.cluster1-pxc-unready.pxc.svc.cluster.local
++ awk '{ print $1 }'
+ LINE_IP=10.233.83.116
+ '[' 10.233.83.116 '!=' 10.233.83.116 ']'
+ read -ra LINE
+ '[' 0 '!=' 0 ']'
+ '[' 0 '!=' 0 ']'
+ CFG=/etc/mysql/node.cnf
++ mysqld -V
++ awk '{print $3}'
++ awk -F. '{print $1"."$2}'
+ MYSQL_VERSION=8.0
+ '[' 8.0 == 8.0 ']'
+ grep -E -q '^[#]?admin-address' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a admin-address=\n' /etc/mysql/node.cnf
+ grep -E -q '^[#]?log_error_suppression_list' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a log_error_suppression_list="MY-010055"\n' /etc/mysql/node.cnf
+ '[' '' == yes ']'
+ grep -E -q '^[#]?wsrep_sst_donor' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a wsrep_sst_donor=\n' /etc/mysql/node.cnf
+ grep -E -q '^[#]?wsrep_node_incoming_address' /etc/mysql/node.cnf
+ grep -E -q '^[#]?wsrep_provider_options' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a wsrep_provider_options="pc.weight=10"\n' /etc/mysql/node.cnf
+ sed -r 's|^[#]?server_id=.*$|server_id=26270930|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?coredumper$|coredumper|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_node_address=.*$|wsrep_node_address=10.233.83.116|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_cluster_name=.*$|wsrep_cluster_name=cluster1-pxc|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_sst_donor=.*$|wsrep_sst_donor=|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_cluster_address=.*$|wsrep_cluster_address=gcomm://|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_node_incoming_address=.*$|wsrep_node_incoming_address=cluster1-pxc-0.cluster1-pxc.pxc.svc.cluster.local:3306|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?admin-address=.*$|admin-address=10.233.83.116|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?extra_max_connections=.*$|extra_max_connections=100|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?extra_port=.*$|extra_port=33062|' /etc/mysql/node.cnf
+ CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+ '[' -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ']'
+ SSL_DIR=/etc/mysql/ssl
+ '[' -f /etc/mysql/ssl/ca.crt ']'
+ CA=/etc/mysql/ssl/ca.crt
+ SSL_INTERNAL_DIR=/etc/mysql/ssl-internal
+ '[' -f /etc/mysql/ssl-internal/ca.crt ']'
+ CA=/etc/mysql/ssl-internal/ca.crt
+ KEY=/etc/mysql/ssl/tls.key
+ CERT=/etc/mysql/ssl/tls.crt
+ '[' -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ']'
+ KEY=/etc/mysql/ssl-internal/tls.key
+ CERT=/etc/mysql/ssl-internal/tls.crt
+ '[' -f /etc/mysql/ssl-internal/ca.crt -a -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ']'
+ sed '/^\[mysqld\]/a pxc-encrypt-cluster-traffic=ON\nssl-ca=/etc/mysql/ssl-internal/ca.crt\nssl-key=/etc/mysql/ssl-internal/tls.key\nssl-cert=/etc/mysql/ssl-internal/tls.crt' /etc/mysql/node.cnf
2024/03/28 01:02:44 Peer finder exiting
Cluster address set to: 
8.0.35-27.1
[mysqld]
pxc-encrypt-cluster-traffic=ON
ssl-ca=/etc/mysql/ssl-internal/ca.crt
ssl-key=/etc/mysql/ssl-internal/tls.key
ssl-cert=/etc/mysql/ssl-internal/tls.crt
wsrep_provider_options="pc.weight=10"

wsrep_sst_donor=

log_error_suppression_list="MY-010055"

admin-address=10.233.83.116

authentication_policy=caching_sha2_password,,
skip_replica_start=ON
enforce-gtid-consistency
gtid-mode=ON
plugin_load="binlog_utils_udf=binlog_utils_udf.so"

datadir=/var/lib/mysql
socket=/tmp/mysql.sock
skip-host-cache

coredumper
server_id=26270930
binlog_format=ROW
default_storage_engine=InnoDB

innodb_flush_log_at_trx_commit  = 0
innodb_flush_method             = O_DIRECT
innodb_file_per_table           = 1
innodb_autoinc_lock_mode=2

bind_address = 0.0.0.0

wsrep_slave_threads=2
wsrep_cluster_address=gcomm://
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so

wsrep_cluster_name=cluster1-pxc
wsrep_node_address=10.233.83.116
wsrep_node_incoming_address=cluster1-pxc-0.cluster1-pxc.pxc.svc.cluster.local:3306

wsrep_sst_method=xtrabackup-v2

[client]
socket=/tmp/mysql.sock

[sst]
cpat=.*\.pem$\|.*init\.ok$\|.*galera\.cache$\|.*wsrep_recovery_verbose\.log$\|.*readiness-check\.sh$\|.*liveness-check\.sh$\|.*get-pxc-state$\|.*sst_in_progress$\|.*sleep-forever$\|.*pmm-prerun\.sh$\|.*sst-xb-tmpdir$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$\|.*pxc-entrypoint\.sh$\|.*unsafe-bootstrap\.sh$\|.*pxc-configure-pxc\.sh\|.*peer-list$\|.*auth_plugin$\|.*version_info$
progress=1

WSREP: Print recovery logs: 
2024-03-28T01:02:45.433971Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-03-28T01:02:45.433991Z 0 [Warning] [MY-011070] [Server] 'binlog_format' is deprecated and will be removed in a future release.
2024-03-28T01:02:45.434011Z 0 [Warning] [MY-011068] [Server] The syntax 'wsrep_slave_threads' is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.
2024-03-28T01:02:45.435410Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2024-03-28T01:02:45.436100Z 0 [Note] [MY-013932] [Server] BuildID[sha1]=bd0342cb308cd633872e6acb9310855ffa230425
2024-03-28T01:02:45.436108Z 0 [Note] [MY-010949] [Server] Basedir set to /usr/.
2024-03-28T01:02:45.436118Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.35-27.1) starting as process 166
2024-03-28T01:02:45.441529Z 0 [Note] [MY-012366] [InnoDB] Using Linux native AIO
2024-03-28T01:02:45.441751Z 0 [Note] [MY-010747] [Server] Plugin 'FEDERATED' is disabled.
2024-03-28T01:02:45.441828Z 0 [Note] [MY-010747] [Server] Plugin 'ndbcluster' is disabled.
2024-03-28T01:02:45.441845Z 0 [Note] [MY-010747] [Server] Plugin 'ndbinfo' is disabled.
2024-03-28T01:02:45.441855Z 0 [Note] [MY-010747] [Server] Plugin 'ndb_transid_mysql_connection_map' is disabled.
2024-03-28T01:02:45.443389Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-03-28T01:02:45.443390Z 1 [Note] [MY-013546] [InnoDB] Atomic write enabled
2024-03-28T01:02:45.443391Z 1 [Note] [MY-012932] [InnoDB] PUNCH HOLE support available
2024-03-28T01:02:45.443392Z 1 [Note] [MY-012944] [InnoDB] Uses event mutexes
2024-03-28T01:02:45.443393Z 1 [Note] [MY-012945] [InnoDB] GCC builtin __atomic_thread_fence() is used for memory barrier
2024-03-28T01:02:45.443394Z 1 [Note] [MY-012948] [InnoDB] Compressed tables use zlib 1.2.13
2024-03-28T01:02:45.446580Z 1 [Note] [MY-012951] [InnoDB] Using hardware accelerated crc32 and polynomial multiplication.
2024-03-28T01:02:45.447319Z 1 [Note] [MY-012203] [InnoDB] Directories to scan './'
2024-03-28T01:02:45.447372Z 1 [Note] [MY-012204] [InnoDB] Scanning './'
2024-03-28T01:02:45.458673Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 9 files.
2024-03-28T01:02:45.460842Z 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 768.000000M, instances = 1, chunk size =128.000000M 
2024-03-28T01:02:45.515270Z 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2024-03-28T01:02:45.521832Z 0 [Note] [MY-011952] [InnoDB] If the mysqld execution user is authorized, page cleaner and LRU manager thread priority can be changed. See the man page of setpriority().
2024-03-28T01:02:45.523434Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2024-03-28T01:02:45.526070Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2024-03-28T01:02:45.551741Z 1 [Note] [MY-013566] [InnoDB] Double write buffer files: 2
2024-03-28T01:02:45.551786Z 1 [Note] [MY-013565] [InnoDB] Double write buffer pages per instance: 4
2024-03-28T01:02:45.551824Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2024-03-28T01:02:45.551857Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2024-03-28T01:02:45.594454Z 1 [Note] [MY-013883] [InnoDB] The latest found checkpoint is at lsn = 36159419 in redo log file ./#innodb_redo/#ib_redo11.
2024-03-28T01:02:45.595337Z 1 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 36158998, whereas checkpoint_lsn = 36159419 and start_lsn = 36158976
2024-03-28T01:02:45.595357Z 1 [Note] [MY-012547] [InnoDB] Log scan progressed past the checkpoint LSN 36158998.
2024-03-28T01:02:45.595367Z 1 [Note] [MY-012551] [InnoDB] Database was not shutdown normally!
2024-03-28T01:02:45.595380Z 1 [Note] [MY-012552] [InnoDB] Starting crash recovery.
2024-03-28T01:02:45.826031Z 1 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 36204847
2024-03-28T01:02:45.840924Z 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2024-03-28T01:02:45.841628Z 1 [Note] [MY-012532] [InnoDB] Applying a batch of 39 redo log records ...
2024-03-28T01:02:45.851625Z 1 [Note] [MY-012533] [InnoDB] 100%
2024-03-28T01:02:46.354019Z 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2024-03-28T01:02:46.411271Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'.
2024-03-28T01:02:46.412641Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'.
2024-03-28T01:02:46.413524Z 1 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.
2024-03-28T01:02:46.413570Z 1 [Note] [MY-011980] [InnoDB] GTID recovery trx_no: 16145
2024-03-28T01:02:46.529692Z 1 [Note] [MY-013776] [InnoDB] Parallel initialization of rseg complete
2024-03-28T01:02:46.529693Z 1 [Note] [MY-013777] [InnoDB] Time taken to initialize rseg using 4 thread: 116130 ms.
2024-03-28T01:02:46.529694Z 1 [Note] [MY-012255] [InnoDB] Removed temporary tablespace data file: "ibtmp1"
2024-03-28T01:02:46.529695Z 1 [Note] [MY-012923] [InnoDB] Creating shared tablespace for temporary tables
2024-03-28T01:02:46.529696Z 1 [Note] [MY-012265] [InnoDB] Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-03-28T01:02:46.836658Z 1 [Note] [MY-012266] [InnoDB] File './ibtmp1' size is now 12 MB.
2024-03-28T01:02:46.836659Z 1 [Note] [MY-013627] [InnoDB] Scanning temp tablespace dir:'./#innodb_temp/'
2024-03-28T01:02:46.948622Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2024-03-28T01:02:46.948623Z 1 [Note] [MY-012976] [InnoDB] Percona XtraDB (http://www.percona.com) 8.0.35-27 started; log sequence number 36204963
2024-03-28T01:02:46.948624Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-03-28T01:02:46.962158Z 1 [Note] [MY-011089] [Server] Data dictionary restarting version '80023'.
2024-03-28T01:02:47.117437Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2024-03-28T01:02:47.120634Z 1 [Note] [MY-012356] [InnoDB] Scanned 11 tablespaces. Validated 11.
2024-03-28T01:02:47.140621Z 1 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2024-03-28T01:02:47.159214Z 1 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2024-03-28T01:02:47.159215Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'.
2024-03-28T01:02:47.167525Z 0 [Note] [MY-011332] [Server] Plugin mysqlx reported: 'IPv6 is available'
2024-03-28T01:02:47.170368Z 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. bind-address: '::' port: 33060'
2024-03-28T01:02:47.170369Z 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. socket: '/var/lib/mysql/mysqlx.sock''
2024-03-28T01:02:47.204621Z 0 [Note] [MY-010902] [Server] Thread priority attribute setting in Resource Group SQL shall be ignored due to unsupported platform or insufficient privilege.
2024-03-28T01:02:47.210107Z 0 [Note] [MY-010855] [Server] Recovering after a crash using binlog
2024-03-28T01:02:47.210108Z 0 [Note] [MY-000000] [WSREP] Before binlog recovery (wsrep position: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:404)
2024-03-28T01:02:47.210109Z 0 [Note] [MY-000000] [WSREP] After binlog recovery (wsrep position: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:404)
2024-03-28T01:02:47.210110Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2024-03-28T01:02:47.221435Z 0 [Note] [MY-013911] [Server] Crash recovery finished in binlog engine. No attempts to commit, rollback or prepare any transactions.
2024-03-28T01:02:47.221436Z 0 [Note] [MY-013911] [Server] Crash recovery finished in InnoDB engine. No attempts to commit, rollback or prepare any transactions.
2024-03-28T01:02:47.221437Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2024-03-28T01:02:47.225008Z 0 [Note] [MY-012487] [InnoDB] DDL log recovery : begin
2024-03-28T01:02:47.225009Z 0 [Note] [MY-012488] [InnoDB] DDL log recovery : end
2024-03-28T01:02:47.225010Z 0 [Warning] [MY-011825] [InnoDB] Skipping buffer pool dump/restore during wsrep recovery.
2024-03-28T01:02:47.237802Z 0 [Note] [MY-012922] [InnoDB] Waiting for purge to start
2024-03-28T01:02:47.305748Z 0 [Note] [MY-000000] [WSREP] Recovered position: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:404
2024-03-28T01:02:47.311549Z 0 [Note] [MY-012330] [InnoDB] FTS optimize thread exiting.
2024-03-28T01:02:48.256681Z 0 [Note] [MY-010120] [Server] Binlog end
2024-03-28T01:02:48.256682Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'binlog_utils_udf'
2024-03-28T01:02:48.256683Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysqlx'
2024-03-28T01:02:48.257945Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysqlx_cache_cleaner'
2024-03-28T01:02:48.257946Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'ngram'
2024-03-28T01:02:48.257947Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'BLACKHOLE'
2024-03-28T01:02:48.257948Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'ARCHIVE'
2024-03-28T01:02:48.257949Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'TempTable'
2024-03-28T01:02:48.257950Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MRG_MYISAM'
2024-03-28T01:02:48.257951Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM'
2024-03-28T01:02:48.257952Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_SESSION_TEMP_TABLESPACES'
2024-03-28T01:02:48.257953Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CACHED_INDEXES'
2024-03-28T01:02:48.257954Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_VIRTUAL'
2024-03-28T01:02:48.257955Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_COLUMNS'
2024-03-28T01:02:48.257956Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLESPACES'
2024-03-28T01:02:48.257957Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_INDEXES'
2024-03-28T01:02:48.257958Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLESTATS'
2024-03-28T01:02:48.257959Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLES'
2024-03-28T01:02:48.257960Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2024-03-28T01:02:48.257961Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2024-03-28T01:02:48.257962Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_CONFIG'
2024-03-28T01:02:48.257963Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2024-03-28T01:02:48.257964Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_DELETED'
2024-03-28T01:02:48.257965Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2024-03-28T01:02:48.257966Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_METRICS'
2024-03-28T01:02:48.257967Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2024-03-28T01:02:48.257968Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2024-03-28T01:02:48.257969Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2024-03-28T01:02:48.257970Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_PAGE'
2024-03-28T01:02:48.257971Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2024-03-28T01:02:48.257972Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2024-03-28T01:02:48.257973Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMPMEM_RESET'
2024-03-28T01:02:48.257974Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMPMEM'
2024-03-28T01:02:48.257975Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_RESET'
2024-03-28T01:02:48.257976Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP'
2024-03-28T01:02:48.257977Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TRX'
2024-03-28T01:02:48.257978Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'InnoDB'
2024-03-28T01:02:48.258662Z 0 [Note] [MY-013072] [InnoDB] Starting shutdown...
2024-03-28T01:02:48.380620Z 0 [Note] [MY-013084] [InnoDB] Log background threads are being closed...
2024-03-28T01:02:48.921846Z 0 [Note] [MY-013854] [InnoDB] Bytes written to disk by DBLWR (ON): 1605632
2024-03-28T01:02:48.921847Z 0 [Note] [MY-012980] [InnoDB] Shutdown completed; log sequence number 36251677
2024-03-28T01:02:48.921848Z 0 [Note] [MY-012255] [InnoDB] Removed temporary tablespace data file: "ibtmp1"
2024-03-28T01:02:48.921849Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MEMORY'
2024-03-28T01:02:48.921850Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV'
2024-03-28T01:02:48.921851Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'PERFORMANCE_SCHEMA'
2024-03-28T01:02:48.921852Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'wsrep'
2024-03-28T01:02:48.921853Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'sha2_cache_cleaner'
2024-03-28T01:02:48.921854Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'caching_sha2_password'
2024-03-28T01:02:48.921855Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'sha256_password'
2024-03-28T01:02:48.921856Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysql_native_password'
2024-03-28T01:02:48.921857Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'daemon_keyring_proxy_plugin'
2024-03-28T01:02:48.924633Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'binlog'
2024-03-28T01:02:48.924634Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.35-27.1)  Percona XtraDB Cluster (GPL), Release rel27, Revision 84d9464, WSREP version 26.1.4.3.
2024-03-28T01:02:47.305748Z 0 [Note] [MY-000000] [WSREP] Recovered position: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:404
+ [[ -z '' ]]
+ [[ -f /var/lib/mysql//grastate.dat ]]
+ [[ 1 != 1 ]]
+ [[ -z '' ]]
+ [[ -f /var/lib/mysql//gvwstate.dat ]]
+ [[ -z '' ]]
+ [[ -f /var/lib/mysql//grastate.dat ]]
+ [[ 1 == 1 ]]
+ [[ -n '' ]]
+ test -e /opt/percona/hookscript/hook.sh
+ init_opt=
+ [[ -f /etc/mysql/init-file/init.sql ]]
+ init_opt=--init-file=/etc/mysql/init-file/init.sql
+ echo 'Using init-file: /etc/mysql/init-file/init.sql'
Using init-file: /etc/mysql/init-file/init.sql
+ exec mysqld --wsrep_start_position=216d94d3-ec04-11ee-8e70-3b9cf1fb681c:404 --init-file=/etc/mysql/init-file/init.sql
2024-03-28T01:03:00.570038Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-03-28T01:03:00.570039Z 0 [Warning] [MY-011070] [Server] 'binlog_format' is deprecated and will be removed in a future release.
2024-03-28T01:03:00.570040Z 0 [Warning] [MY-011068] [Server] The syntax 'wsrep_slave_threads' is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.
2024-03-28T01:03:00.570041Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2024-03-28T01:03:00.572630Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.35-27.1) starting as process 1
2024-03-28T01:03:00.576625Z 0 [Warning] [MY-010068] [Server] CA certificate /etc/mysql/ssl-internal/ca.crt is self signed.
2024-03-28T01:03:00.576626Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-03-28T01:03:00.577228Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/galera4/libgalera_smm.so initial position: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:404
2024-03-28T01:03:00.577229Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/galera4/libgalera_smm.so'
2024-03-28T01:03:00.577524Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.16(b73532f) by Codership Oy <info@codership.com> (modified by Percona <https://percona.com/>) loaded successfully.
2024-03-28T01:03:00.578041Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
2024-03-28T01:03:00.578203Z 0 [Warning] [MY-000000] [Galera] SSL compression is not effective. The option socket.ssl_compression is deprecated and will be removed in future releases.
2024-03-28T01:03:00.578204Z 0 [Warning] [MY-000000] [Galera] Parameter 'socket.ssl_compression' is deprecated and will be removed in future versions
2024-03-28T01:03:00.578477Z 0 [Note] [MY-000000] [Galera] Found saved state: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:-1, safe_to_bootstrap: 1
2024-03-28T01:03:00.578828Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c
Seqno: -1 - -1
Offset: -1
Synced: 0
EncVersion: 1
Encrypted: 0
MasterKeyConst UUID: 216a2e41-ec04-11ee-a13b-f250810a8f1f
MasterKey UUID: 00000000-0000-0000-0000-000000000000
MasterKey ID: 0
2024-03-28T01:03:00.578917Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c, offset: -1
2024-03-28T01:03:00.578932Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
2024-03-28T01:03:00.672617Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2024-03-28T01:03:00.672618Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-405
2024-03-28T01:03:00.676063Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...  0.0% (     0/189272 bytes) complete.
2024-03-28T01:03:00.676064Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (189272/189272 bytes) complete.
2024-03-28T01:03:00.676065Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 185/590 locked buffers
2024-03-28T01:03:00.676066Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 134067360/134217728
2024-03-28T01:03:00.677439Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 10.233.83.116; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 10; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = /etc/mysql/ssl-internal/ca.crt; socket.ssl_cert = /etc/mysql/ssl-internal/tls.crt; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = /etc/mysql/ssl-internal/tls.key; socket.ssl_reload = 1; 
2024-03-28T01:03:00.693744Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-03-28T01:03:00.693745Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:404, protocol version: -1
2024-03-28T01:03:00.695389Z 0 [Note] [MY-000000] [WSREP] Starting replication
2024-03-28T01:03:00.695390Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1
2024-03-28T01:03:00.695391Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:404
2024-03-28T01:03:00.695392Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2024-03-28T01:03:00.695733Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2024-03-28T01:03:00.695734Z 0 [Note] [MY-000000] [Galera] backend: asio
2024-03-28T01:03:00.696621Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 
2024-03-28T01:03:00.696796Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2024-03-28T01:03:00.696797Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2024-03-28T01:03:00.696958Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2024-03-28T01:03:00.697093Z 0 [Note] [MY-000000] [Galera] (ec354ca0-afa5, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2024-03-28T01:03:00.697094Z 0 [Note] [MY-000000] [Galera] (ec354ca0-afa5, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2024-03-28T01:03:00.697254Z 0 [Note] [MY-000000] [Galera] EVS version 1
2024-03-28T01:03:00.697477Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group 'cluster1-pxc'
2024-03-28T01:03:00.697478Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery
2024-03-28T01:03:00.697638Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2024-03-28T01:03:00.697757Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2024-03-28T01:03:00.697758Z 0 [Note] [MY-000000] [Galera] Node ec354ca0-afa5 state primary
2024-03-28T01:03:00.697759Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,ec354ca0-afa5,1)
memb {
        ec354ca0-afa5,0
        }
joined {
        }
left {
        }
partitioned {
        }
)
2024-03-28T01:03:00.697760Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-03-28T01:03:00.699435Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2024-03-28T01:03:00.699436Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2024-03-28T01:03:00.700368Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2024-03-28T01:03:00.700369Z 0 [Note] [MY-000000] [Galera] Opened channel 'cluster1-pxc'
2024-03-28T01:03:00.700534Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2024-03-28T01:03:00.700698Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: ec35e507-ec9e-11ee-92b2-6b47cea6aa7d
2024-03-28T01:03:00.700699Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: ec35e507-ec9e-11ee-92b2-6b47cea6aa7d
2024-03-28T01:03:00.700700Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: ec35e507-ec9e-11ee-92b2-6b47cea6aa7d from 0 (cluster1-pxc-0)
2024-03-28T01:03:00.700701Z 0 [Note] [MY-000000] [Galera] Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 0,
        members    = 1/1 (primary/total),
        act_id     = 404,
        last_appl. = 404,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 216d94d3-ec04-11ee-8e70-3b9cf1fb681c
2024-03-28T01:03:00.700848Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-03-28T01:03:00.700867Z 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (405)
2024-03-28T01:03:00.700868Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2024-03-28T01:03:00.700869Z 0 [Note] [MY-000000] [Galera] Member 0.0 (cluster1-pxc-0) synced with group.
2024-03-28T01:03:00.700919Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 405)
2024-03-28T01:03:00.700929Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2024-03-28T01:03:00.700930Z 2 [Note] [MY-000000] [Galera] ####### processing CC 405, local, ordered
2024-03-28T01:03:00.700931Z 2 [Note] [MY-000000] [Galera] Maybe drain monitors from 404 upto current CC event 405 upto:404
2024-03-28T01:03:00.700932Z 2 [Note] [MY-000000] [Galera] Drain monitors from 404 up to 404
2024-03-28T01:03:00.700933Z 2 [Note] [MY-000000] [Galera] Process first view: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c my uuid: ec354ca0-ec9e-11ee-afa5-670b7468cea8
2024-03-28T01:03:00.700934Z 2 [Note] [MY-000000] [Galera] Server cluster1-pxc-0 connected to cluster at position 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:405 with ID ec354ca0-ec9e-11ee-afa5-670b7468cea8
2024-03-28T01:03:00.700935Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2024-03-28T01:03:00.700936Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-28T01:03:00.700937Z 2 [Note] [MY-000000] [Galera] ####### My UUID: ec354ca0-ec9e-11ee-afa5-670b7468cea8
2024-03-28T01:03:00.700938Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no
2024-03-28T01:03:00.700939Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-03-28T01:03:00.700940Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-03-28T01:03:00.700941Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-03-28T01:03:00.700942Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 405
2024-03-28T01:03:00.700943Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-03-28T01:03:00.700944Z 2 [Note] [MY-000000] [Galera] ================================================
View:
  id: 216d94d3-ec04-11ee-8e70-3b9cf1fb681c:405
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: ec354ca0-ec9e-11ee-afa5-670b7468cea8, cluster1-pxc-0
=================================================
2024-03-28T01:03:00.700945Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2024-03-28T01:03:00.700946Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-28T01:03:00.700947Z 2 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing
2024-03-28T01:03:00.700948Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-28T01:03:00.706444Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-03-28T01:03:01.228624Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-03-28T01:03:01.422884Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2024-03-28T01:03:01.447657Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2024-03-28T01:03:01.633130Z 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_admin. See below for the description of exact issue.
2024-03-28T01:03:01.633131Z 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2024-03-28T01:03:01.633132Z 0 [System] [MY-013603] [Server] No TLS configuration was given for channel mysql_admin; re-using TLS configuration of channel mysql_main.
2024-03-28T01:03:01.636621Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/lib/mysql' in the path is accessible to all OS users. Consider choosing a different directory.
2024-03-28T01:03:01.643239Z 0 [ERROR] [MY-000067] [Server] unknown variable 'audit_log_file=/var/lib/mysql/audit.log'.
2024-03-28T01:03:01.643345Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-03-28T01:03:01.643369Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2024-03-28T01:03:03.650579Z 0 [Note] [MY-000000] [WSREP] Server status change initializing -> disconnecting
2024-03-28T01:03:03.650580Z 1 [Note] [MY-000000] [WSREP] rollbacker thread exiting 1
2024-03-28T01:03:03.650581Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-28T01:03:03.650731Z 0 [Note] [MY-000000] [Galera] Closing send monitor...
2024-03-28T01:03:03.650732Z 0 [Note] [MY-000000] [Galera] Closed send monitor.
2024-03-28T01:03:03.650733Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2024-03-28T01:03:03.650734Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2024-03-28T01:03:03.650837Z 2 [ERROR] [MY-000000] [Galera] Exception: State wait was interrupted
2024-03-28T01:03:03.650920Z 2 [ERROR] [MY-000000] [Galera] View callback failed. This is unrecoverable, restart required. (FATAL)
         at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.35/percona-xtradb-cluster-galera/galera/src/replicator_smm.cpp:submit_view_info():2616
2024-03-28T01:03:03.650940Z 2 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()
2024-03-28T01:03:03.650957Z 2 [Note] [MY-000000] [Galera] mysqld: Terminated.
2024-03-28T01:03:03Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=bd0342cb308cd633872e6acb9310855ffa230425
Server Version: 8.0.35-27.1 Percona XtraDB Cluster (GPL), Release rel27, Revision 84d9464, WSREP version 26.1.4.3, wsrep_26.1.4.3

Thread pointer: 0x7f97a8000b60
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
2024-03-28T01:03:03.650998Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend
stack_bottom = 7f97c42f8c30 thread_stack 0x100000
2024-03-28T01:03:03.651053Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2024-03-28T01:03:03.651054Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2024-03-28T01:03:03.651055Z 0 [Note] [MY-000000] [Galera] gcomm: closed
2024-03-28T01:03:03.651056Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2024-03-28T01:03:03.651057Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2024-03-28T01:03:03.651058Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2024-03-28T01:03:03.651059Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> CLOSED (TO: 405)
2024-03-28T01:03:03.651060Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2024-03-28T01:03:03.651061Z 0 [Note] [MY-000000] [Galera] recv_thread() joined.
2024-03-28T01:03:03.651062Z 0 [Note] [MY-000000] [Galera] Closing replication queue.
2024-03-28T01:03:03.651063Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x227d851]
/usr/sbin/mysqld(print_fatal_signal(int)+0x39f) [0x12802df]
/usr/sbin/mysqld(handle_fatal_signal+0xd8) [0x12803c8]
/lib64/libpthread.so.0(+0x12cf0) [0x7f97dce7acf0]
/lib64/libc.so.6(abort+0x203) [0x7f97db1fbf81]
/usr/lib64/galera4/libgalera_smm.so(+0x435db) [0x7f97ce5a25db]
/usr/lib64/galera4/libgalera_smm.so(+0x68882) [0x7f97ce5c7882]
/usr/lib64/galera4/libgalera_smm.so(+0x1cbdc) [0x7f97ce57bbdc]
/usr/lib64/galera4/libgalera_smm.so(+0x7822a) [0x7f97ce5d722a]
/usr/lib64/galera4/libgalera_smm.so(+0x787ae) [0x7f97ce5d77ae]
/usr/lib64/galera4/libgalera_smm.so(+0x9c722) [0x7f97ce5fb722]
/usr/lib64/galera4/libgalera_smm.so(+0x7691b) [0x7f97ce5d591b]
/usr/lib64/galera4/libgalera_smm.so(+0x4cdf2) [0x7f97ce5abdf2]
/usr/sbin/mysqld(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*)+0x12) [0x2c66ca2]
/usr/sbin/mysqld() [0x12d263d]
/usr/sbin/mysqld(start_wsrep_THD+0x394) [0xf712d4]
/usr/sbin/mysqld() [0x27397e9]
/lib64/libpthread.so.0(+0x81ca) [0x7f97dce701ca]
/lib64/libc.so.6(clone+0x43) [0x7f97db213e73]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 2
Status: KILL_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file using lib coredumper
PATH: (null)
lanss315425 commented 7 months ago

@MAXxATTAXx How did you handle it