Closed luckydogxf closed 2 years ago
sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/kick OVN_Southbound f8be
#
failed to send removal request ovs-appctl: /var/run/ovn/ovnsb_db.ctl: server returned an error'
Guess it's because the cluster has no leader, so stale members cannot be kicked out.
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
To add a server to a cluster, run ovsdb-tool join-cluster on the new server and start ovsdb-server. To remove a running server from a cluster, use ovs-appctl to invoke the cluster/leave command. When a server fails and cannot be recovered, e.g. because its hard disk crashed, or to otherwise remove a server that is down from a cluster, use ovs-appctl to invoke cluster/kick to make the remaining servers kick it out of the cluster.
The above methods for adding and removing servers only work for healthy clusters, that is, for clusters with no more failures than their maximum tolerance. For example, in a 3-server cluster, the failure of 2 servers prevents servers joining or leaving the cluster (as well as database access).
cluster/kick
to force stale
IPs out because the cluster is unhealthy now.85 and 86 are stale IP and cannot be back. Currently the cluster isn't established. So is there any way to modify ovs-db manually ? I have to rebuild the cluster from scratch if not.
It's an openstack OVN-* component, so I believe I can recover OVSDB-NB From Neutron DB and OVSDB-SB could be synced from compute nodes, right ?
On Fri, Jan 21, 2022 at 9:16 PM Ilya Maximets @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/242#issuecomment-1018495636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANT2BZXMXERKMGSTUD7D6ELUXFMBDANCNFSM5MOUYYAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster? ----> I know this is strange, Let me explain what happened.
juju
, and
checked it was unhealthy. we removed that node later( IP may be 85).It happens.
On Sat, Jan 22, 2022 at 9:39 AM luckydog xf @.***> wrote:
Thanks for your reply. I read this docs( https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#clustered-database-service-model )
To add a server to a cluster, run ovsdb-tool join-cluster on the new server and start ovsdb-server. To remove a running server from a cluster, use ovs-appctl to invoke the cluster/leave command. When a server fails and cannot be recovered, e.g. because its hard disk crashed, or to otherwise remove a server that is down from a cluster, use ovs-appctl to invoke cluster/kick to make the remaining servers kick it out of the cluster.
The above methods for adding and removing servers only work for healthy clusters, that is, for clusters with no more failures than their maximum tolerance. For example, in a 3-server cluster, the failure of 2 servers prevents servers joining or leaving the cluster (as well as database access).
Just realize that I cannot use commands like
cluster/kick
to force stale IPs out because the cluster is unhealthy now.85 and 86 are stale IP and cannot be back. Currently the cluster isn't established. So is there any way to modify ovs-db manually ? I have to rebuild the cluster from scratch if not.
It's an openstack OVN-* component, so I believe I can recover OVSDB-NB From Neutron DB and OVSDB-SB could be synced from compute nodes, right ?
On Fri, Jan 21, 2022 at 9:16 PM Ilya Maximets @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/242#issuecomment-1018495636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANT2BZXMXERKMGSTUD7D6ELUXFMBDANCNFSM5MOUYYAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
Seems it's impossible to change OVSDB-SB and OVSDB-NB databases manually. I have to stop all ovn- services, remove /var/lib/ovn/.db and then start them.
On Sat, Jan 22, 2022 at 9:47 AM luckydog xf @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster? ----> I know this is strange, Let me explain what happened.
- One of three ovn nodes failed, we added a new one by
juju
, and checked it was unhealthy. we removed that node later( IP may be 85).- We repeated #1 step twice , this time the IP may be 86 and 124.
- The three-newly-added nodes are removed later. .
On Sat, Jan 22, 2022 at 9:39 AM luckydog xf @.***> wrote:
Thanks for your reply. I read this docs( https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#clustered-database-service-model )
To add a server to a cluster, run ovsdb-tool join-cluster on the new server and start ovsdb-server. To remove a running server from a cluster, use ovs-appctl to invoke the cluster/leave command. When a server fails and cannot be recovered, e.g. because its hard disk crashed, or to otherwise remove a server that is down from a cluster, use ovs-appctl to invoke cluster/kick to make the remaining servers kick it out of the cluster.
The above methods for adding and removing servers only work for healthy clusters, that is, for clusters with no more failures than their maximum tolerance. For example, in a 3-server cluster, the failure of 2 servers prevents servers joining or leaving the cluster (as well as database access).
Just realize that I cannot use commands like
cluster/kick
to force stale IPs out because the cluster is unhealthy now.85 and 86 are stale IP and cannot be back. Currently the cluster isn't established. So is there any way to modify ovs-db manually ? I have to rebuild the cluster from scratch if not.
It's an openstack OVN-* component, so I believe I can recover OVSDB-NB From Neutron DB and OVSDB-SB could be synced from compute nodes, right ?
On Fri, Jan 21, 2022 at 9:16 PM Ilya Maximets @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/242#issuecomment-1018495636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANT2BZXMXERKMGSTUD7D6ELUXFMBDANCNFSM5MOUYYAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
Currently I have three nodes, IP 147/133/69
# /etc/default/ovn-central
OVN_CTL_OPTS= \
--db-nb-cluster-local-addr=172.16.200.147 \
--db-nb-cluster-local-port=6643 \
--db-nb-cluster-local-proto=ssl \
--ovn-nb-db-ssl-key=/etc/ovn/key_host \
--ovn-nb-db-ssl-cert=/etc/ovn/cert_host \
--ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \
--db-nb-cluster-remote-addr=172.16.200.133 \
--db-nb-cluster-remote-port=6643 \
--db-nb-cluster-remote-proto=ssl \
--db-sb-cluster-local-addr=172.16.200.147 \
--db-sb-cluster-local-port=6644 \
--db-sb-cluster-local-proto=ssl \
--ovn-sb-db-ssl-key=/etc/ovn/key_host \
--ovn-sb-db-ssl-cert=/etc/ovn/cert_host \
--ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \
--db-sb-cluster-remote-addr=172.16.200.133 \
--db-sb-cluster-remote-port=6644 \
--db-sb-cluster-remote-proto=ssl
### 200.133 ###
OVN_CTL_OPTS= \
--db-nb-cluster-local-addr=172.16.200.133 \
--db-nb-cluster-local-port=6643 \
--db-nb-cluster-local-proto=ssl \
--ovn-nb-db-ssl-key=/etc/ovn/key_host \
--ovn-nb-db-ssl-cert=/etc/ovn/cert_host \
--ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \
--db-nb-cluster-remote-addr=172.16.200.147 \
--db-nb-cluster-remote-port=6643 \
--db-nb-cluster-remote-proto=ssl \
--db-sb-cluster-local-addr=172.16.200.133 \
--db-sb-cluster-local-port=6644 \
--db-sb-cluster-local-proto=ssl \
--ovn-sb-db-ssl-key=/etc/ovn/key_host \
--ovn-sb-db-ssl-cert=/etc/ovn/cert_host \
--ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \
--db-sb-cluster-remote-addr=172.16.200.147 \
--db-sb-cluster-remote-port=6644 \
--db-sb-cluster-remote-proto=ssl
### 200.69 ###
OVN_CTL_OPTS= \
--db-nb-cluster-local-addr=172.16.200.69 \
--db-nb-cluster-local-port=6643 \
--db-nb-cluster-local-proto=ssl \
--ovn-nb-db-ssl-key=/etc/ovn/key_host \
--ovn-nb-db-ssl-cert=/etc/ovn/cert_host \
--ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \
--db-nb-cluster-remote-addr=172.16.200.147 \
--db-nb-cluster-remote-port=6643 \
--db-nb-cluster-remote-proto=ssl \
--db-sb-cluster-local-addr=172.16.200.69 \
--db-sb-cluster-local-port=6644 \
--db-sb-cluster-local-proto=ssl \
--ovn-sb-db-ssl-key=/etc/ovn/key_host \
--ovn-sb-db-ssl-cert=/etc/ovn/cert_host \
--ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \
--db-sb-cluster-remote-addr=172.16.200.147 \
--db-sb-cluster-remote-port=6644 \
--db-sb-cluster-remote-proto=ssl
Are we can see, 147 tries to connect to 133, and 133 tries to connect to 147, while 69 tries to connect to 147.
147 --->133
133 --> 147
69 ---> 147
I stopped all ovn- services, and removed /var/lib/ovn/ovn and /var/lib/ovn/.ovn. Then restarted all ovn- services in three nodes.
But the cluster wasn't established, please see below information.
# status of 147 ##
sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound
f6fe
Name: OVN_Southbound
Cluster ID: not yet known
Server ID: f6fe (f6fe41de-2ac6-4a48-a748-b263dd4509d5)
Address: ssl:172.16.200.147:6644
Status: joining cluster
Remotes for joining: ssl:172.16.200.133:6644
Role: follower
Term: 0
Leader: unknown
Vote: unknown
Connections: ->3c7c <-3c7c
# status of 133 ###
Name: OVN_Southbound
Cluster ID: not yet known
Server ID: 3c7c (3c7cedef-2e85-49bf-bb04-5fffab7f2a58)
Address: ssl:172.16.200.133:6644
Status: joining cluster
Remotes for joining: ssl:172.16.200.147:6644
# satus of 69 ###
Name: OVN_Southbound
Cluster ID: not yet known
Server ID: ae8b (ae8b7cbf-f333-45a3-bb1c-819b75bd48dc)
Address: ssl:172.16.200.69:6644
Status: joining cluster
Remotes for joining: ssl:172.16.200.147:6644
Role: follower
Term: 0
Leader: unknown
Vote: unknown
====
So what's the correct way to bring them up again ? Thanks.
On Sat, Jan 22, 2022 at 9:56 AM luckydog xf @.***> wrote:
Seems it's impossible to change OVSDB-SB and OVSDB-NB databases manually. I have to stop all ovn- services, remove /var/lib/ovn/.db and then start them.
On Sat, Jan 22, 2022 at 9:47 AM luckydog xf @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster? ----> I know this is strange, Let me explain what happened.
- One of three ovn nodes failed, we added a new one by
juju
, and checked it was unhealthy. we removed that node later( IP may be 85).- We repeated #1 step twice , this time the IP may be 86 and 124.
- The three-newly-added nodes are removed later. .
On Sat, Jan 22, 2022 at 9:39 AM luckydog xf @.***> wrote:
Thanks for your reply. I read this docs( https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#clustered-database-service-model )
To add a server to a cluster, run ovsdb-tool join-cluster on the new server and start ovsdb-server. To remove a running server from a cluster, use ovs-appctl to invoke the cluster/leave command. When a server fails and cannot be recovered, e.g. because its hard disk crashed, or to otherwise remove a server that is down from a cluster, use ovs-appctl to invoke cluster/kick to make the remaining servers kick it out of the cluster.
The above methods for adding and removing servers only work for healthy clusters, that is, for clusters with no more failures than their maximum tolerance. For example, in a 3-server cluster, the failure of 2 servers prevents servers joining or leaving the cluster (as well as database access).
Just realize that I cannot use commands like
cluster/kick
to force stale IPs out because the cluster is unhealthy now.85 and 86 are stale IP and cannot be back. Currently the cluster isn't established. So is there any way to modify ovs-db manually ? I have to rebuild the cluster from scratch if not.
It's an openstack OVN-* component, so I believe I can recover OVSDB-NB From Neutron DB and OVSDB-SB could be synced from compute nodes, right ?
On Fri, Jan 21, 2022 at 9:16 PM Ilya Maximets @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/242#issuecomment-1018495636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANT2BZXMXERKMGSTUD7D6ELUXFMBDANCNFSM5MOUYYAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
It seems there must exist an initial node in the cluster. Otherwise it becomes a chicken egg problem
.
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.147 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ ########### REMOVE BELOW LINE ############################
--db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.147 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ ########### REMOVE BELOW LINE ############################
--db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
@.***:/var/lib/ovn$ sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound 5bc0 Name: OVN_Southbound Cluster ID: 9e33 (9e336acc-b7d6-4541-8ec2-16a6dc8133cf) Server ID: 5bc0 (5bc09c15-df7e-4629-9650-2e188a406864) Address: ssl:172.16.200.147:6644 Status: cluster member Role: follower Term: 2 Leader: acea Vote: unknown
Election timer: 1000 Log: [2, 9] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->0000 ->105b <-acea <-105b Servers: acea (acea at ssl:172.16.200.133:6644) 5bc0 (5bc0 at ssl:172.16.200.147:6644) (self) 105b (105b at ssl:172.16.200.69:6644) @.***:/var/lib/ovn$ sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound 9707 Name: OVN_Northbound Cluster ID: 825d (825dda4c-b654-4e8c-8e15-a2648928efb9) Server ID: 9707 (97070892-50c2-483c-96e2-9998e3788f56) Address: ssl:172.16.200.147:6643 Status: cluster member Role: follower Term: 2 Leader: 112e Vote: unknown
Election timer: 1000 Log: [2, 9] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->0000 ->0000 <-112e <-954b Servers: 9707 (9707 at ssl:172.16.200.147:6643) (self) 112e (112e at ssl:172.16.200.69:6643) 954b (954b at ssl:172.16.200.133:6643)
On Sat, Jan 22, 2022 at 6:46 PM luckydog xf @.***> wrote:
Currently I have three nodes, IP 147/133/69
200.147 OVN-* Config.
/etc/default/ovn-central
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.147 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.133 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.147 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.133 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
200.133
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.133 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.147 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.133 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.147 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
200.69
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.69 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.147 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.69 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.147 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
Are we can see, 147 tries to connect to 133, and 133 tries to connect to 147, while 69 tries to connect to 147.
147 --->133 133 --> 147 69 ---> 147
I stopped all ovn- services, and removed /var/lib/ovn/ovn and /var/lib/ovn/.ovn. Then restarted all ovn- services in three nodes.
But the cluster wasn't established, please see below information.
status of 147
sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound f6fe Name: OVN_Southbound Cluster ID: not yet known Server ID: f6fe (f6fe41de-2ac6-4a48-a748-b263dd4509d5) Address: ssl:172.16.200.147:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.133:6644 Role: follower Term: 0 Leader: unknown Vote: unknown
Connections: ->3c7c <-3c7c
status of 133
Name: OVN_Southbound Cluster ID: not yet known Server ID: 3c7c (3c7cedef-2e85-49bf-bb04-5fffab7f2a58) Address: ssl:172.16.200.133:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.147:6644
satus of 69
Name: OVN_Southbound Cluster ID: not yet known Server ID: ae8b (ae8b7cbf-f333-45a3-bb1c-819b75bd48dc) Address: ssl:172.16.200.69:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.147:6644 Role: follower Term: 0 Leader: unknown Vote: unknown
So what's the correct them to bring them up again ? Thanks.
On Sat, Jan 22, 2022 at 9:56 AM luckydog xf @.***> wrote:
Seems it's impossible to change OVSDB-SB and OVSDB-NB databases manually. I have to stop all ovn- services, remove /var/lib/ovn/.db and then start them.
On Sat, Jan 22, 2022 at 9:47 AM luckydog xf @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster? ----> I know this is strange, Let me explain what happened.
- One of three ovn nodes failed, we added a new one by
juju
, and checked it was unhealthy. we removed that node later( IP may be 85).- We repeated #1 step twice , this time the IP may be 86 and 124.
- The three-newly-added nodes are removed later. .
On Sat, Jan 22, 2022 at 9:39 AM luckydog xf @.***> wrote:
Thanks for your reply. I read this docs( https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#clustered-database-service-model )
To add a server to a cluster, run ovsdb-tool join-cluster on the new server and start ovsdb-server. To remove a running server from a cluster, use ovs-appctl to invoke the cluster/leave command. When a server fails and cannot be recovered, e.g. because its hard disk crashed, or to otherwise remove a server that is down from a cluster, use ovs-appctl to invoke cluster/kick to make the remaining servers kick it out of the cluster.
The above methods for adding and removing servers only work for healthy clusters, that is, for clusters with no more failures than their maximum tolerance. For example, in a 3-server cluster, the failure of 2 servers prevents servers joining or leaving the cluster (as well as database access).
Just realize that I cannot use commands like
cluster/kick
to force stale IPs out because the cluster is unhealthy now.85 and 86 are stale IP and cannot be back. Currently the cluster isn't established. So is there any way to modify ovs-db manually ? I have to rebuild the cluster from scratch if not.
It's an openstack OVN-* component, so I believe I can recover OVSDB-NB From Neutron DB and OVSDB-SB could be synced from compute nodes, right ?
On Fri, Jan 21, 2022 at 9:16 PM Ilya Maximets @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/242#issuecomment-1018495636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANT2BZXMXERKMGSTUD7D6ELUXFMBDANCNFSM5MOUYYAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
root 7509 1 0 11:19 ? 00:00:00 ovn-northd -vconsole:emer -vsyslog:err -vfile:info --ovnnb-db=ssl:172.16.200.147:6641 ,ssl:172.16.200.133:6641,ssl:172.16.200.69:6641 --ovnsb-db=ssl: 172.16.200.147:16642,ssl:172.16.200.133:16642,ssl:172.16.200.69:16642 -c /etc/ovn/cert_host -C /etc/ovn/ovn-central.crt -p /etc/ovn/key_host --no-chdir --log-file=/var/log/ovn/ovn-northd.log --pidfile=/var/run/ovn/ovn-northd.pid --detach
All are running on three nodes.
But 6641 and 6642 are not up.
sudo netstat -lpan | grep 6641 sudo netstat -lpan | grep 6642
sudo ovn-sbctl get-connection # read-write role="" pssl:16642 read-write role="ovn-controller" pssl:6642
Turns out I need to allow connections.
However,
sudo ovn-sbctl set-connection role="" pssl:16642 if I add the second one, the first one would be overwritten.
Weird.
On Sat, Jan 22, 2022 at 7:29 PM luckydog xf @.***> wrote:
- Stopped ovn- and removed /var/lib/ovn of 147.
- modify /etc/default/ovn-central
200.147 OVN-* Config.
/etc/default/ovn-central
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.147 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ ########### REMOVE BELOW LINE ############################
--db-nb-cluster-remote-addr=172.16.200.133 \
--db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.147 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ ########### REMOVE BELOW LINE ############################
--db-sb-cluster-remote-addr=172.16.200.133 \
--db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
Then started ovn- in 147, 133 and 69, cluster is back now. Use ovc-appctl cluster/kick to remove 147, stop ovn-, revert /etc/default/ovn-central of 147, remove /var/lib/ovn/*. Finally restart. Now 147 is completely back to normal.
@.***:/var/lib/ovn$ sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound 5bc0 Name: OVN_Southbound Cluster ID: 9e33 (9e336acc-b7d6-4541-8ec2-16a6dc8133cf) Server ID: 5bc0 (5bc09c15-df7e-4629-9650-2e188a406864) Address: ssl:172.16.200.147:6644 Status: cluster member Role: follower Term: 2 Leader: acea Vote: unknown
Election timer: 1000 Log: [2, 9] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->0000 ->105b <-acea <-105b Servers: acea (acea at ssl:172.16.200.133:6644) 5bc0 (5bc0 at ssl:172.16.200.147:6644) (self) 105b (105b at ssl:172.16.200.69:6644) @.***:/var/lib/ovn$ sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound 9707 Name: OVN_Northbound Cluster ID: 825d (825dda4c-b654-4e8c-8e15-a2648928efb9) Server ID: 9707 (97070892-50c2-483c-96e2-9998e3788f56) Address: ssl:172.16.200.147:6643 Status: cluster member Role: follower Term: 2 Leader: 112e Vote: unknown
Election timer: 1000 Log: [2, 9] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->0000 ->0000 <-112e <-954b Servers: 9707 (9707 at ssl:172.16.200.147:6643) (self) 112e (112e at ssl:172.16.200.69:6643) 954b (954b at ssl:172.16.200.133:6643)
On Sat, Jan 22, 2022 at 6:46 PM luckydog xf @.***> wrote:
Currently I have three nodes, IP 147/133/69
200.147 OVN-* Config.
/etc/default/ovn-central
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.147 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.133 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.147 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.133 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
200.133
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.133 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.147 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.133 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.147 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
200.69
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.69 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.147 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.69 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.147 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
Are we can see, 147 tries to connect to 133, and 133 tries to connect to 147, while 69 tries to connect to 147.
147 --->133 133 --> 147 69 ---> 147
I stopped all ovn- services, and removed /var/lib/ovn/ovn and /var/lib/ovn/.ovn. Then restarted all ovn- services in three nodes.
But the cluster wasn't established, please see below information.
status of 147
sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound f6fe Name: OVN_Southbound Cluster ID: not yet known Server ID: f6fe (f6fe41de-2ac6-4a48-a748-b263dd4509d5) Address: ssl:172.16.200.147:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.133:6644 Role: follower Term: 0 Leader: unknown Vote: unknown
Connections: ->3c7c <-3c7c
status of 133
Name: OVN_Southbound Cluster ID: not yet known Server ID: 3c7c (3c7cedef-2e85-49bf-bb04-5fffab7f2a58) Address: ssl:172.16.200.133:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.147:6644
satus of 69
Name: OVN_Southbound Cluster ID: not yet known Server ID: ae8b (ae8b7cbf-f333-45a3-bb1c-819b75bd48dc) Address: ssl:172.16.200.69:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.147:6644 Role: follower Term: 0 Leader: unknown Vote: unknown
So what's the correct them to bring them up again ? Thanks.
On Sat, Jan 22, 2022 at 9:56 AM luckydog xf @.***> wrote:
Seems it's impossible to change OVSDB-SB and OVSDB-NB databases manually. I have to stop all ovn- services, remove /var/lib/ovn/.db and then start them.
On Sat, Jan 22, 2022 at 9:47 AM luckydog xf @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster? ----> I know this is strange, Let me explain what happened.
- One of three ovn nodes failed, we added a new one by
juju
, and checked it was unhealthy. we removed that node later( IP may be 85).- We repeated #1 step twice , this time the IP may be 86 and 124.
- The three-newly-added nodes are removed later. .
On Sat, Jan 22, 2022 at 9:39 AM luckydog xf @.***> wrote:
Thanks for your reply. I read this docs( https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#clustered-database-service-model )
To add a server to a cluster, run ovsdb-tool join-cluster on the new server and start ovsdb-server. To remove a running server from a cluster, use ovs-appctl to invoke the cluster/leave command. When a server fails and cannot be recovered, e.g. because its hard disk crashed, or to otherwise remove a server that is down from a cluster, use ovs-appctl to invoke cluster/kick to make the remaining servers kick it out of the cluster.
The above methods for adding and removing servers only work for healthy clusters, that is, for clusters with no more failures than their maximum tolerance. For example, in a 3-server cluster, the failure of 2 servers prevents servers joining or leaving the cluster (as well as database access).
Just realize that I cannot use commands like
cluster/kick
to force stale IPs out because the cluster is unhealthy now.85 and 86 are stale IP and cannot be back. Currently the cluster isn't established. So is there any way to modify ovs-db manually ? I have to rebuild the cluster from scratch if not.
It's an openstack OVN-* component, so I believe I can recover OVSDB-NB From Neutron DB and OVSDB-SB could be synced from compute nodes, right ?
On Fri, Jan 21, 2022 at 9:16 PM Ilya Maximets < @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/242#issuecomment-1018495636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANT2BZXMXERKMGSTUD7D6ELUXFMBDANCNFSM5MOUYYAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
@.***:~$ sudo ovn-sbctl set-connection read-write role="" pssl:16642
@.***:~$ sudo ovn-sbctl set-connection read-write role="ovn-controller" pssl:6642
@.***:~$ sudo ovn-sbctl get-connection read-write role="ovn-controller" pssl:6642
weird. The first one is overwided.
On Sat, Jan 22, 2022 at 8:35 PM luckydog xf @.***> wrote:
However OVSDB SB and NB don't work.
root 7509 1 0 11:19 ? 00:00:00 ovn-northd -vconsole:emer -vsyslog:err -vfile:info --ovnnb-db=ssl:172.16.200.147:6641 ,ssl:172.16.200.133:6641,ssl:172.16.200.69:6641 --ovnsb-db=ssl: 172.16.200.147:16642,ssl:172.16.200.133:16642,ssl:172.16.200.69:16642 -c /etc/ovn/cert_host -C /etc/ovn/ovn-central.crt -p /etc/ovn/key_host --no-chdir --log-file=/var/log/ovn/ovn-northd.log --pidfile=/var/run/ovn/ovn-northd.pid --detach
All are running on three nodes.
But 6641 and 6642 are not up.
Both are empty.
sudo netstat -lpan | grep 6641 sudo netstat -lpan | grep 6642
sudo ovn-sbctl get-connection # read-write role="" pssl:16642 read-write role="ovn-controller" pssl:6642
Just find that I need to allow connections.
However,
sudo ovn-sbctl set-connection role="" pssl:16642 if I add the second one, the first one would be override.
Weird.
On Sat, Jan 22, 2022 at 7:29 PM luckydog xf @.***> wrote:
- Stopped ovn- and removed /var/lib/ovn of 147.
- modify /etc/default/ovn-central
200.147 OVN-* Config.
/etc/default/ovn-central
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.147 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ ########### REMOVE BELOW LINE ############################
--db-nb-cluster-remote-addr=172.16.200.133 \
--db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.147 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ ########### REMOVE BELOW LINE ############################
--db-sb-cluster-remote-addr=172.16.200.133 \
--db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
Then started ovn- in 147, 133 and 69, cluster is back now. Use ovc-appctl cluster/kick to remove 147, stop ovn-, revert /etc/default/ovn-central of 147, remove /var/lib/ovn/*. Finally restart. Now 147 is completely back to normal.
@.***:/var/lib/ovn$ sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound 5bc0 Name: OVN_Southbound Cluster ID: 9e33 (9e336acc-b7d6-4541-8ec2-16a6dc8133cf) Server ID: 5bc0 (5bc09c15-df7e-4629-9650-2e188a406864) Address: ssl:172.16.200.147:6644 Status: cluster member Role: follower Term: 2 Leader: acea Vote: unknown
Election timer: 1000 Log: [2, 9] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->0000 ->105b <-acea <-105b Servers: acea (acea at ssl:172.16.200.133:6644) 5bc0 (5bc0 at ssl:172.16.200.147:6644) (self) 105b (105b at ssl:172.16.200.69:6644) @.***:/var/lib/ovn$ sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound 9707 Name: OVN_Northbound Cluster ID: 825d (825dda4c-b654-4e8c-8e15-a2648928efb9) Server ID: 9707 (97070892-50c2-483c-96e2-9998e3788f56) Address: ssl:172.16.200.147:6643 Status: cluster member Role: follower Term: 2 Leader: 112e Vote: unknown
Election timer: 1000 Log: [2, 9] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->0000 ->0000 <-112e <-954b Servers: 9707 (9707 at ssl:172.16.200.147:6643) (self) 112e (112e at ssl:172.16.200.69:6643) 954b (954b at ssl:172.16.200.133:6643)
On Sat, Jan 22, 2022 at 6:46 PM luckydog xf @.***> wrote:
Currently I have three nodes, IP 147/133/69
200.147 OVN-* Config.
/etc/default/ovn-central
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.147 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.133 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.147 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.133 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
200.133
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.133 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.147 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.133 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.147 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
200.69
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.69 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.147 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.69 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.147 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
Are we can see, 147 tries to connect to 133, and 133 tries to connect to 147, while 69 tries to connect to 147.
147 --->133 133 --> 147 69 ---> 147
I stopped all ovn- services, and removed /var/lib/ovn/ovn and /var/lib/ovn/.ovn. Then restarted all ovn- services in three nodes.
But the cluster wasn't established, please see below information.
status of 147
sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound f6fe Name: OVN_Southbound Cluster ID: not yet known Server ID: f6fe (f6fe41de-2ac6-4a48-a748-b263dd4509d5) Address: ssl:172.16.200.147:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.133:6644 Role: follower Term: 0 Leader: unknown Vote: unknown
Connections: ->3c7c <-3c7c
status of 133
Name: OVN_Southbound Cluster ID: not yet known Server ID: 3c7c (3c7cedef-2e85-49bf-bb04-5fffab7f2a58) Address: ssl:172.16.200.133:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.147:6644
satus of 69
Name: OVN_Southbound Cluster ID: not yet known Server ID: ae8b (ae8b7cbf-f333-45a3-bb1c-819b75bd48dc) Address: ssl:172.16.200.69:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.147:6644 Role: follower Term: 0 Leader: unknown Vote: unknown
So what's the correct them to bring them up again ? Thanks.
On Sat, Jan 22, 2022 at 9:56 AM luckydog xf @.***> wrote:
Seems it's impossible to change OVSDB-SB and OVSDB-NB databases manually. I have to stop all ovn- services, remove /var/lib/ovn/.db and then start them.
On Sat, Jan 22, 2022 at 9:47 AM luckydog xf @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster? ----> I know this is strange, Let me explain what happened.
- One of three ovn nodes failed, we added a new one by
juju
, and checked it was unhealthy. we removed that node later( IP may be 85).- We repeated #1 step twice , this time the IP may be 86 and 124.
- The three-newly-added nodes are removed later. .
On Sat, Jan 22, 2022 at 9:39 AM luckydog xf @.***> wrote:
Thanks for your reply. I read this docs( https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#clustered-database-service-model )
To add a server to a cluster, run ovsdb-tool join-cluster on the new server and start ovsdb-server. To remove a running server from a cluster, use ovs-appctl to invoke the cluster/leave command. When a server fails and cannot be recovered, e.g. because its hard disk crashed, or to otherwise remove a server that is down from a cluster, use ovs-appctl to invoke cluster/kick to make the remaining servers kick it out of the cluster.
The above methods for adding and removing servers only work for healthy clusters, that is, for clusters with no more failures than their maximum tolerance. For example, in a 3-server cluster, the failure of 2 servers prevents servers joining or leaving the cluster (as well as database access).
Just realize that I cannot use commands like
cluster/kick
to force stale IPs out because the cluster is unhealthy now.85 and 86 are stale IP and cannot be back. Currently the cluster isn't established. So is there any way to modify ovs-db manually ? I have to rebuild the cluster from scratch if not.
It's an openstack OVN-* component, so I believe I can recover OVSDB-NB From Neutron DB and OVSDB-SB could be synced from compute nodes, right ?
On Fri, Jan 21, 2022 at 9:16 PM Ilya Maximets < @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/242#issuecomment-1018495636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANT2BZXMXERKMGSTUD7D6ELUXFMBDANCNFSM5MOUYYAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
Damn it, just run # sudo ovn-sbctl set-connection read-write role="" pssl:16642 read-write role="ovn-controller" pssl:6642
On Sat, Jan 22, 2022 at 9:00 PM luckydog xf @.***> wrote:
@.***:~$ sudo ovn-sbctl set-connection read-write role="" pssl:16642
@.***:~$ sudo ovn-sbctl set-connection read-write role="ovn-controller" pssl:6642
@.***:~$ sudo ovn-sbctl get-connection read-write role="ovn-controller" pssl:6642
weird. The first one is overwided.
On Sat, Jan 22, 2022 at 8:35 PM luckydog xf @.***> wrote:
However OVSDB SB and NB don't work.
root 7509 1 0 11:19 ? 00:00:00 ovn-northd -vconsole:emer -vsyslog:err -vfile:info --ovnnb-db=ssl: 172.16.200.147:6641,ssl:172.16.200.133:6641,ssl:172.16.200.69:6641 --ovnsb-db=ssl:172.16.200.147:16642,ssl:172.16.200.133:16642,ssl: 172.16.200.69:16642 -c /etc/ovn/cert_host -C /etc/ovn/ovn-central.crt -p /etc/ovn/key_host --no-chdir --log-file=/var/log/ovn/ovn-northd.log --pidfile=/var/run/ovn/ovn-northd.pid --detach
All are running on three nodes.
But 6641 and 6642 are not up.
Both are empty.
sudo netstat -lpan | grep 6641 sudo netstat -lpan | grep 6642
sudo ovn-sbctl get-connection # read-write role="" pssl:16642 read-write role="ovn-controller" pssl:6642
Just find that I need to allow connections.
However,
sudo ovn-sbctl set-connection role="" pssl:16642 if I add the second one, the first one would be override.
Weird.
On Sat, Jan 22, 2022 at 7:29 PM luckydog xf @.***> wrote:
- Stopped ovn- and removed /var/lib/ovn of 147.
- modify /etc/default/ovn-central
200.147 OVN-* Config.
/etc/default/ovn-central
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.147 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ ########### REMOVE BELOW LINE ############################
--db-nb-cluster-remote-addr=172.16.200.133 \
--db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.147 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ ########### REMOVE BELOW LINE ############################
--db-sb-cluster-remote-addr=172.16.200.133 \
--db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
Then started ovn- in 147, 133 and 69, cluster is back now. Use ovc-appctl cluster/kick to remove 147, stop ovn-, revert /etc/default/ovn-central of 147, remove /var/lib/ovn/*. Finally restart. Now 147 is completely back to normal.
@.***:/var/lib/ovn$ sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound 5bc0 Name: OVN_Southbound Cluster ID: 9e33 (9e336acc-b7d6-4541-8ec2-16a6dc8133cf) Server ID: 5bc0 (5bc09c15-df7e-4629-9650-2e188a406864) Address: ssl:172.16.200.147:6644 Status: cluster member Role: follower Term: 2 Leader: acea Vote: unknown
Election timer: 1000 Log: [2, 9] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->0000 ->105b <-acea <-105b Servers: acea (acea at ssl:172.16.200.133:6644) 5bc0 (5bc0 at ssl:172.16.200.147:6644) (self) 105b (105b at ssl:172.16.200.69:6644) @.***:/var/lib/ovn$ sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound 9707 Name: OVN_Northbound Cluster ID: 825d (825dda4c-b654-4e8c-8e15-a2648928efb9) Server ID: 9707 (97070892-50c2-483c-96e2-9998e3788f56) Address: ssl:172.16.200.147:6643 Status: cluster member Role: follower Term: 2 Leader: 112e Vote: unknown
Election timer: 1000 Log: [2, 9] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->0000 ->0000 <-112e <-954b Servers: 9707 (9707 at ssl:172.16.200.147:6643) (self) 112e (112e at ssl:172.16.200.69:6643) 954b (954b at ssl:172.16.200.133:6643)
On Sat, Jan 22, 2022 at 6:46 PM luckydog xf @.***> wrote:
Currently I have three nodes, IP 147/133/69
200.147 OVN-* Config.
/etc/default/ovn-central
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.147 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.133 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.147 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.133 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
200.133
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.133 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.147 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.133 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.147 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
200.69
OVN_CTL_OPTS= \ --db-nb-cluster-local-addr=172.16.200.69 \ --db-nb-cluster-local-port=6643 \ --db-nb-cluster-local-proto=ssl \ --ovn-nb-db-ssl-key=/etc/ovn/key_host \ --ovn-nb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-nb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-nb-cluster-remote-addr=172.16.200.147 \ --db-nb-cluster-remote-port=6643 \ --db-nb-cluster-remote-proto=ssl \ --db-sb-cluster-local-addr=172.16.200.69 \ --db-sb-cluster-local-port=6644 \ --db-sb-cluster-local-proto=ssl \ --ovn-sb-db-ssl-key=/etc/ovn/key_host \ --ovn-sb-db-ssl-cert=/etc/ovn/cert_host \ --ovn-sb-db-ssl-ca-cert=/etc/ovn/ovn-central.crt \ --db-sb-cluster-remote-addr=172.16.200.147 \ --db-sb-cluster-remote-port=6644 \ --db-sb-cluster-remote-proto=ssl
Are we can see, 147 tries to connect to 133, and 133 tries to connect to 147, while 69 tries to connect to 147.
147 --->133 133 --> 147 69 ---> 147
I stopped all ovn- services, and removed /var/lib/ovn/ovn and /var/lib/ovn/.ovn. Then restarted all ovn- services in three nodes.
But the cluster wasn't established, please see below information.
status of 147
sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound f6fe Name: OVN_Southbound Cluster ID: not yet known Server ID: f6fe (f6fe41de-2ac6-4a48-a748-b263dd4509d5) Address: ssl:172.16.200.147:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.133:6644 Role: follower Term: 0 Leader: unknown Vote: unknown
Connections: ->3c7c <-3c7c
status of 133
Name: OVN_Southbound Cluster ID: not yet known Server ID: 3c7c (3c7cedef-2e85-49bf-bb04-5fffab7f2a58) Address: ssl:172.16.200.133:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.147:6644
satus of 69
Name: OVN_Southbound Cluster ID: not yet known Server ID: ae8b (ae8b7cbf-f333-45a3-bb1c-819b75bd48dc) Address: ssl:172.16.200.69:6644 Status: joining cluster Remotes for joining: ssl:172.16.200.147:6644 Role: follower Term: 0 Leader: unknown Vote: unknown
So what's the correct them to bring them up again ? Thanks.
On Sat, Jan 22, 2022 at 9:56 AM luckydog xf @.***> wrote:
Seems it's impossible to change OVSDB-SB and OVSDB-NB databases manually. I have to stop all ovn- services, remove /var/lib/ovn/.db and then start them.
On Sat, Jan 22, 2022 at 9:47 AM luckydog xf @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster? ----> I know this is strange, Let me explain what happened.
- One of three ovn nodes failed, we added a new one by
juju
, and checked it was unhealthy. we removed that node later( IP may be 85).- We repeated #1 step twice , this time the IP may be 86 and 124.
- The three-newly-added nodes are removed later. .
On Sat, Jan 22, 2022 at 9:39 AM luckydog xf @.***> wrote:
Thanks for your reply. I read this docs( https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#clustered-database-service-model )
To add a server to a cluster, run ovsdb-tool join-cluster on the new server and start ovsdb-server. To remove a running server from a cluster, use ovs-appctl to invoke the cluster/leave command. When a server fails and cannot be recovered, e.g. because its hard disk crashed, or to otherwise remove a server that is down from a cluster, use ovs-appctl to invoke cluster/kick to make the remaining servers kick it out of the cluster.
The above methods for adding and removing servers only work for healthy clusters, that is, for clusters with no more failures than their maximum tolerance. For example, in a 3-server cluster, the failure of 2 servers prevents servers joining or leaving the cluster (as well as database access).
Just realize that I cannot use commands like
cluster/kick
to force stale IPs out because the cluster is unhealthy now.85 and 86 are stale IP and cannot be back. Currently the cluster isn't established. So is there any way to modify ovs-db manually ? I have to rebuild the cluster from scratch if not.
It's an openstack OVN-* component, so I believe I can recover OVSDB-NB From Neutron DB and OVSDB-SB could be synced from compute nodes, right ?
On Fri, Jan 21, 2022 at 9:16 PM Ilya Maximets < @.***> wrote:
If you had only 3 nodes, how did you end up with 5 servers in the ovsdb cluster?
On the topic, you have a 5-server cluster now with only 2 of them being alive. Therefore, there is no quorum, hence there is no way to elect a leader, hence no way to add or delete servers or perform any other database modifications. You need to bring back to live one of the dead servers (not rebuild them), otherwise the cluster is effectively unrecoverable without manual changes in the database files, which would be very intrusive and hard to do correctly. If you can not bring up one of the dead servers, the easiest option would be to destroy the current cluster all together and re-build from scratch.
— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/242#issuecomment-1018495636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANT2BZXMXERKMGSTUD7D6ELUXFMBDANCNFSM5MOUYYAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
The following thing is setting connection for NB(6641) and using neutron-ovn-db-sync-util
to sync data from neutron database to OVN-NB, and OVN-SB would be populated soon after restarting ovn-* on compute nodes.
Problem resolved. Thanks.
@luckydogxf I'm glad you resolved the issue. And thank you for documenting the steps here, this might be really helpful for someone else looking around for solutions.
I guess, the bottom line here is: It's important to kick the dead server (with ovs-appctl cluster/kick) from the cluster, before replacing it.
I have three ovn cluster nodes, the first one is
172.16.200.147
, the second one172.16.200.133
, and the third one was broken and just has been rebuilt with ip172.16.200.63
.But cluster has no NB and SB leader, here are status of
############ 172.16.200.147 ##############
172.16.200.133
172.16.200.63, the newly build one
As we can see there are two unused nodes ( 200.86 and 200.85). 147 and 133 do not elect properly, so there is no leader for ovs-nb and ovs-sb database. And the newly built one isn't join the cluster
I try to use
' sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/kick OVN_Southbound f8be
failed to send removal request ovs-appctl: /var/run/ovn/ovnsb_db.ctl: server returned an error'
to kick out 200.85 and 200.86 on 200.133 and 200.147, they just don't work.
As we can see, the connection is wrong.
133 of NB/SB.
133 voted itself, and it becomes
candiate
, and Let's check 147147 votes b550, which is 133, too. It becomes
follower
.So the main problem is 'Connections', 147 and 133 try to connect 85 and 86 ( stale IPs). Somehow it makes the election void.