Scenario:

2 nodes A and B, edge client running on node C
1 edge client with initial connection node A + it's volume owned by A
write data to the volume
block edge connection towards node A (e.g. iptables drop)
edge client should reconnect to node B
continue writing data
now block connection between 2 voldrvs

First part of this test is similar to: #493

Previous mechanism used redirection timeout to steal the volume. Now a new keepalive mechanism is introduced which should trigger this.

New volumedriver parameters related to this testcase:

All 4 params below are dynamically reconfigurable:

volume_router/vrouter_keepalive_time_secs: interval with which to check liveliness, default for now: 60
vrouter_keepalive_interval_secs: after a probe was sent, check after that many secs if a response was received / retry if not. default for now: 10
vrouter_keepalive_retries: number of retries before considering the other side dead

Avoid triggering the vrouter_redirect_timeout_ms which defaults to 5000 ms, set this to a high number (e.g. 300000)

Advise by Arne: Redirect timeout should be increased, and the keepalive settings decreased (the defaults will probably be changed when merging this stuff) IOW; for testing I'd suggest to set the redirect_timeout_ms to a very high value so it does not fire and to use smaller values for the keepalive, so it fires earlier

Cover area

volumedriver HA

User story

Running volume via edge client should not be affected/experience disconnects/timeout which the test flow is triggered.

Potential problem that might occur

Volume goes in readonly, timeouts occur, ...

Optionally try to finetune values to a working set which can be returned to volumedriver team to include as defaults

Additional information

Setup

Hyperconverged - 3 node setup
vpool on node A and B
node C can be used to run edge client

openvstorage / integrationtests