Sharding is introduce to scale out the experiments.
Indeed ombt-controller is a bottleneck as it needs to handle a large
broadcast domain and receive results from all the agents. The
control-plane of the experiment can be sharded: several separated
control paths can be created each one of them consist in one
ombt-controller and a control bus (currently rabbitmq) and a subset of
the agents of the experiment.
Sharding is test_case specific so sharding is done explicitly. It
consists in generating each control-plane shard and the configuration of
the agents of the shard. All of these configuration is then merged and
passed to Ansible to start the agents.
Things to know:
There is an extra effort to do when
aggregating the results as it comes from several controllers.
The agents of two different shards can handle different number of
messages, this somehow mislead the controller when it waits the end of
the experiment. Currently some controllers (this whose agents receive
less messages than expected) wait until a time out is triggered. This
doen't hinder the correctness of the experiment but lead to extra time
spent at the end of an experiment.
Sharding is introduce to scale out the experiments.
Indeed ombt-controller is a bottleneck as it needs to handle a large broadcast domain and receive results from all the agents. The control-plane of the experiment can be sharded: several separated control paths can be created each one of them consist in one ombt-controller and a control bus (currently rabbitmq) and a subset of the agents of the experiment.
Sharding is test_case specific so sharding is done explicitly. It consists in generating each control-plane shard and the configuration of the agents of the shard. All of these configuration is then merged and passed to Ansible to start the agents.
Things to know: