cppforlife / turbulence-release

Turbulence release is used for injecting failure scenarios into any BOSH deployment.
Apache License 2.0
49 stars 15 forks source link

tc error when using Loss and Delay with ControlNet #21

Open mattiekat opened 6 years ago

mattiekat commented 6 years ago

I am consistently getting the error RTNETLINK answers: File exists when I attempt to use a ControlNet incident with both Delay and Loss. Furthermore, after this configuration is used, the error continues to happen even for ControlNet incidents which have only one or the other and have been tested to work on their own otherwise.

The following configuration consistently yields this problem.

{
 "Tasks": [{
   "Type": "ControlNet",
   "Timeout": "30s",
   "Delay": "100ms",
   "Loss": "30%"
 }],

 "Selector": {
    "Deployment": {
        "Name": "cf"
    },
    "Group": {
        "Name": "diego-cell"
    }
 }
}

The full error reported is:

Task execution: Shelling out to tc to add packet loss: Running command: 'tc qdisc add dev silk-vtep root netem loss 30% 75%', stdout: '', stderr: 'RTNETLINK answers: File exists
': exit status 2

When using ping -f -c 500 <DIEGO-CELL-IP> I find that it does introduce some latency which does not seem to go away but it does not introduce any packet loss.

This issue does not occur if the incidents are separated as such:

{
 "Tasks": [{
   "Type": "ControlNet",
   "Timeout": "30s",
   "Delay": "100ms"
 },{
   "Type": "ControlNet",
   "Timeout": "30s",
   "Loss": "30%"
 }],

 "Selector": {
    "Deployment": {
        "Name": "cf"
    },
    "Group": {
        "Name": "diego-cell"
    }
 }
}

The error also occurs if Loss and LossCorrelation are specified without Delay.