search

worstcase / blockade

Docker-based utility for testing network failures and partitions in distributed applications
http://blockade.readthedocs.org
Apache License 2.0
907 stars 72 forks source link

Chaining of Commands e.g. flaky + slow #80

Open mkotzerke opened 5 years ago

mkotzerke commented 5 years ago

As far as i understand this isnt yet implemented but should be technically possible. Do you see something that would prevent this ? I may be able to contribute

labisso commented 5 years ago

Hi, thanks for your comment, and your patience with my slow reply. I think chaining of commands should be possible and easy to implement, but I am having a hard time picturing what the CLI structure would look like. Do you have some ideas?

Another possibility is just to chain different commands in a wrapper shell script (or via the REST API). That was my original intent with the Blockade CLI.

mkotzerke commented 5 years ago

Hi, thanks for the reply. My idea for the CLI was something like this:

blockade flaky --add --all blockade duplicate --add --all blockade slow --add --all blockade flaky --rm --all blockade duplicate --rm mycontainer1

The Result would be all container being slow and all but "mycontainer1" also duplicating packages To keep track of this i would keep a dict for every container with its options:

mycontainer1:{duplicate=False, slow=True,flaky=False} mycontainer2:{duplicate=True, slow=True,flaky=False} mycontainer3:{duplicate=True, slow=True,flaky=False}

It would also make sense to have a way to apply all network options(flaky,slow,..) defined in a config file or by the Rest API at once.

I would also have to test which commands can be chained, because maybe its not possible to chain every command together. What do you think about it ?

labisso commented 5 years ago

I'm not sure I understand your CLI example. Are those separate calls to blockade? If so that should already work. Blockade is already stateful - just the state about flaky/duplicated/etc is stored in the iptables or tc config of each container.

mkotzerke commented 5 years ago

Yes They are separate calls and you are right that blockade is already stateful. Sorry for explaining it so confusingly.

But as it is right now i cant have packet drop, duplicate packages and latency at the same time for a container. This is what i try to implement. For that i would have to keep track of multiple states (eg flaky and slow at the same time)

labisso commented 5 years ago

Oh! I understand what you mean now. I think that is a useful feature, and I'd certainly welcome a pull request! I think it'd be best to avoid directly storing any additional state about the containers in blockade's internal state file (.blockade/). This is to avoid cases where blockade's idea of a container's state gets out of sync with reality. If we calculate the state directly from the source of truth (tc) everytime, there is less chance of bugs.

As far as the implementation, the key code to figure out is the TrafficControl class in net.py. Right now with every change we fully reset the qdisc state, but we'd need to make it respect your new --add option to be additive instead. And also the network_state() function needs to support detecting cases where the network is both e.g. slow and flaky. This would also need to be appropriately passed up through the CLI/API, so that blockade status still works.