search

codership / galera

Synchronous multi-master replication library
GNU General Public License v2.0
448 stars 176 forks source link

Unable to bootstrap cluster by setting `pc.bootstrap`: `1 (Operation not permitted)` #584

Open Dav1dde opened 3 years ago

Dav1dde commented 3 years ago

MariaDB: 10.5.6

In a 3-Node cluster after all nodes lost connection due to a network outage, the recovery with SET GLOBAL wsrep_provider_options="pc.bootstrap=1"; (same with 'yes' and 'true') fails with:

MariaDB Client:

MariaDB [(none)]> SET GLOBAL wsrep_provider_options="pc.bootstrap=1";
ERROR 1210 (HY000): Incorrect arguments to SET

Log:

error setting param pc.bootstrap to value true: : 1 (Operation not permitted)

Another attempt:

2020-11-09 10:37:37 176839 [Note] WSREP: refresh_provider_options: pc.bootstrap=YES
2020-11-09 10:37:37 176839 [ERROR] WSREP: Set options returned 7
2020-11-09 10:37:37 176839 [Warning] WSREP: Unknown parameter 'pc.bootstrap'
     at gcomm/src/pc_proto.cpp:set_param():1682
2020-11-09 10:37:37 176839 [Warning] WSREP: error setting param pc.bootstrap to value YES: : 1 (Operation not permitted)
2020-11-09 10:37:37 176839 [Warning] WSREP: 3f9e407c-9dee sending install message failed: Resource temporarily unavailable

The same error happens on all 3 nodes.

Status from one of the nodes:

MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_%';
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name                 | Value                                                                                                                                          |
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| wsrep_local_state_uuid        | 4594209d-18fc-11eb-9681-b28f02f627d9                                                                                                           |
| wsrep_protocol_version        | 10                                                                                                                                             |
| wsrep_last_committed          | 258606                                                                                                                                         |
| wsrep_replicated              | 40761                                                                                                                                          |
| wsrep_replicated_bytes        | 31724264                                                                                                                                       |
| wsrep_repl_keys               | 176825                                                                                                                                         |
| wsrep_repl_keys_bytes         | 2392864                                                                                                                                        |
| wsrep_repl_data_bytes         | 26558976                                                                                                                                       |
| wsrep_repl_other_bytes        | 0                                                                                                                                              |
| wsrep_received                | 94921                                                                                                                                          |
| wsrep_received_bytes          | 129938288                                                                                                                                      |
| wsrep_local_commits           | 39450                                                                                                                                          |
| wsrep_local_cert_failures     | 646                                                                                                                                            |
| wsrep_local_replays           | 0                                                                                                                                              |
| wsrep_local_send_queue        | 0                                                                                                                                              |
| wsrep_local_send_queue_max    | 1                                                                                                                                              |
| wsrep_local_send_queue_min    | 0                                                                                                                                              |
| wsrep_local_send_queue_avg    | 0                                                                                                                                              |
| wsrep_local_recv_queue        | 0                                                                                                                                              |
| wsrep_local_recv_queue_max    | 3                                                                                                                                              |
| wsrep_local_recv_queue_min    | 0                                                                                                                                              |
| wsrep_local_recv_queue_avg    | 0.000389798                                                                                                                                    |
| wsrep_local_cached_downto     | 153461                                                                                                                                         |
| wsrep_flow_control_paused_ns  | 34712660                                                                                                                                       |
| wsrep_flow_control_paused     | 6.71535e-08                                                                                                                                    |
| wsrep_flow_control_sent       | 0                                                                                                                                              |
| wsrep_flow_control_recv       | 3                                                                                                                                              |
| wsrep_cert_deps_distance      | 22.1808                                                                                                                                        |
| wsrep_apply_oooe              | 0.0144375                                                                                                                                      |
| wsrep_apply_oool              | 8.5711e-05                                                                                                                                     |
| wsrep_apply_window            | 1.01444                                                                                                                                        |
| wsrep_commit_oooe             | 0                                                                                                                                              |
| wsrep_commit_oool             | 0                                                                                                                                              |
| wsrep_commit_window           | 1.01289                                                                                                                                        |
| wsrep_local_state             | 0                                                                                                                                              |
| wsrep_local_state_comment     | Initialized                                                                                                                                    |
| wsrep_cert_index_size         | 9                                                                                                                                              |
| wsrep_causal_reads            | 1302                                                                                                                                           |
| wsrep_cert_interval           | 181.193                                                                                                                                        |
| wsrep_open_transactions       | 0                                                                                                                                              |
| wsrep_open_connections        | 0                                                                                                                                              |
| wsrep_incoming_addresses      | AUTO                                                                                                                                           |
| wsrep_cluster_weight          | 0                                                                                                                                              |
| wsrep_desync_count            | 0                                                                                                                                              |
| wsrep_evs_delayed             |                                                                                                                                                |
| wsrep_evs_evict_list          |                                                                                                                                                |
| wsrep_evs_repl_latency        | 6.502e-06/6.502e-06/6.502e-06/0/1                                                                                                              |
| wsrep_evs_state               | GATHER                                                                                                                                         |
| wsrep_gcomm_uuid              | ae819cf5-1dc1-11eb-84fb-0a8279be2a05                                                                                                           |
| wsrep_applier_thread_count    | 1                                                                                                                                              |
| wsrep_cluster_capabilities    |                                                                                                                                                |
| wsrep_cluster_conf_id         | 18446744073709551615                                                                                                                           |
| wsrep_cluster_size            | 1                                                                                                                                              |
| wsrep_cluster_state_uuid      | 4594209d-18fc-11eb-9681-b28f02f627d9                                                                                                           |
| wsrep_cluster_status          | non-Primary                                                                                                                                    |
| wsrep_connected               | ON                                                                                                                                             |
| wsrep_local_bf_aborts         | 687                                                                                                                                            |
| wsrep_local_index             | 0                                                                                                                                              |
| wsrep_provider_capabilities   | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: |
| wsrep_provider_name           | Galera                                                                                                                                         |
| wsrep_provider_vendor         | Codership Oy <info@codership.com>                                                                                                              |
| wsrep_provider_version        | 26.4.5(rb3764ab6)                                                                                                                              |
| wsrep_ready                   | OFF                                                                                                                                            |
| wsrep_rollbacker_thread_count | 1                                                                                                                                              |
| wsrep_thread_count            | 2                                                                                                                                              |
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+

Node 1 and 3 had wsrep_last_committed = 258606 the 2nd node had wsrep_last_committed = 258603.

A full recovery with shutting down the nodes and setting safe_to_bootstrap eventually worked, after multiple segfaults from Node 2.