Validate voldrv HA redirection timeout does not trigger stealing

[pploegaert]

Scenario:

• prerequisites: -- RORA should not be enabled -- reads must come from the backend via proxy not from the scocache -- use SYNC DTL

• 2 nodes A and B, edge client running on node C

• 1 edge client with initial connection node A + it's volume owned by A

• write data to the volume

• block edge connection towards node A (e.g. iptables drop)

• edge client should reconnect to node B

• continue writing data

• redirect should now occur from voldrv on node B -> voldrv on node A

• stop/kill the proxy

• wait for the redirection timeout to occur

• verify no stealing occurs

• verify an I/O error occurs after the timeout

• start the proxy

• reads should now be possible again

First part of this test is similar to: #493 Previous mechanism used redirection timeout to steal the volume. Now a new keepalive mechanism is introduced which should trigger this. Hence the redirection timeout should only give an I/O error not result in volume stealing

Volumedriver parameters related to this testcase - all are dynamically reconfigurable:

• vrouter_redirect_timeout_ms: defaults to 5000 ms
• volume_router/vrouter_keepalive_time_secs: interval with which to check liveliness, default for now: 60
• vrouter_keepalive_interval_secs: after a probe was sent, check after that many secs if a response was received / retry if not. default for now: 10
• vrouter_keepalive_retries: number of retries before considering the other side dead. default for now: 5s

Cover area

volumedriver HA

Additional information

Setup

• Hyperconverged - 3 node setup
• vpool on node A and B
• node C can be used to run edge client

[JeffreyDevloo]

Hmmm looking at this ticket more in detail and thinking about the implementation I have some concerns/ questions:

• When the redirect has occurred, you want the proxy to be killed on the node where the io is being rerouted. wouldn't this cut off all remaining IO causing the stealing threshhold to not be reached?
• An IO error will always occur with this tests due to the reason specified above
• When restarting the proxy and you get IO back, you are back to testing the redirect. Keep in mind that when the error would occur, your system will most likely go into RO mode or sth like that so by default the IO will just magically appear again.

@pploegaert Any insights?

[pploegaert]

New suggested defaults: vrouter_redirect_timeout > vrouter_keepalive_time + vrouter_keepalive_interval_secs vrouter_keepalive_retries ? e.g. 120s > 15s + 2 5s = 25s