openvswitch / ovs-issues

Issue tracker repo for Open vSwitch
10 stars 3 forks source link

OVSDB Relay `ovsdb-server/set-relay-source-probe-interval` not fully changing default probe interval #320

Closed legitYosal closed 5 months ago

legitYosal commented 5 months ago

Hello, After connecting 10 hypervisors to 5 relays and increasing number of logical_flows to 400K, in order to test high load on cluster, We have encountered this error on relays:

|reconnect|ERR|tcp:172.25.2.1:14222: no response to inactivity probe after 5 seconds, disconnecting

After upgrading relays from version 2.17 to 3.2.1 which supports ovsdb-server/set-relay-source-probe-interval, again tested the cluster with setting probe interval to 60000, which resulted in the connections from relay -> SB not dropping, but connections from ovn-controller -> relay are again droping after 5 seconds, it seems setting interval with the following command:

ovs-appctl -t /var/run/openvswitch/ovsdb-server.17.ctl ovsdb-server/set-relay-source-probe-interval 60000

Did not change the default value on all functions! also ovn-controller interval is set to 60 seconds and the issue is coming from relay not ovn-controller.

Is there any patch on this or any way to change default probe by editing the code(where are default values)?

igsilya commented 5 months ago

Hi @legitYosal . The set-relay-source-probe-interval command only sets probe interval for relay-to-source connection and only in that particular direction. If you want to configure probes for relay-to-controller direction, you need to configure it on the remote on which the relay is listening. This can be done via database, in case you configure the relay remote via the database, i.e. --remote=db:DB:Table:.... This should be easy enough if you're able to re-use the same remote as the Southbound DB. You may just use --remote=db:OVN_Southbound,SB_Global,connections.

If you can't re-use the Sb connections because, for example, you need a relay to listen on a different IP or a different port, that becomes a little more complex. In this case the only way to configure the probe interval will be to create another database with a Local_Config schema (https://www.openvswitch.org/support/dist-docs/ovsdb.local-config.5.html) and store connections there. There are also no handy tools for manipulating this database, so you'll need to write transactions directly with ovsdb-client or some other tool.

Upcoming OVS 3.3 release will bring in the --config-file option that will make the life much easier: https://patchwork.ozlabs.org/project/openvswitch/cover/20240109225142.1987981-1-i.maximets@ovn.org/ . There is an example with a remote configuration as well as relay server in the cover letter. You'll need to use not yet released branch-3.3 for this to work. Official release of OVS 3.3 is in about 2 weeks.

legitYosal commented 5 months ago

Thank you @igsilya , very helpful, I would close this as it is not a bug