The ability to manage the launching of workers in a K8s enviroment is covered by #1430.
One of the pains of caliper in a multiple VM environment is that you need to have to go onto each worker VM and launch some caliper remote workers before the benchmark will run. You need to know how many workers are needed as well as the number of optimal workers per VM (a VM will be optimally used if it runs a specific number of workers controlled by the number of cores. This is because a node process utilises only a single core for the execution of the javascript code, although this could be changing for node and I know it does use other threads for things such as garbage collection but can be discounted).
It would be good if a manager could handle the remote launching of these remote workers. One way could be to use ssh, but maybe another option could be to use a server communicating either via REST or over MQTT (if MQTT is used then we would have to have the ability to enable security on the MQTT broker and have caliper support that).
This caliper remote worker controller should be able to
report the number of cores available
launch a set of remote workers (if any are idle then it's a top up action maybe to handle the case where not all workers were used)
The caliper manager should split the work across all known remote worker controllers based on an weight configuration of the number of cores each remote VM has. A default weight would be 1.5 (eg a 4 core VM would have optimally 6 workers).
If the sum of workers across the VMs is less than the number of workers requested, then the caliper manager can split the remaining workers equally across the VMs and display a warning to say it has done this.
The ability to manage the launching of workers in a K8s enviroment is covered by #1430. One of the pains of caliper in a multiple VM environment is that you need to have to go onto each worker VM and launch some caliper remote workers before the benchmark will run. You need to know how many workers are needed as well as the number of optimal workers per VM (a VM will be optimally used if it runs a specific number of workers controlled by the number of cores. This is because a node process utilises only a single core for the execution of the javascript code, although this could be changing for node and I know it does use other threads for things such as garbage collection but can be discounted).
It would be good if a manager could handle the remote launching of these remote workers. One way could be to use ssh, but maybe another option could be to use a server communicating either via REST or over MQTT (if MQTT is used then we would have to have the ability to enable security on the MQTT broker and have caliper support that).
This caliper remote worker controller should be able to
The caliper manager should split the work across all known remote worker controllers based on an weight configuration of the number of cores each remote VM has. A default weight would be 1.5 (eg a 4 core VM would have optimally 6 workers). If the sum of workers across the VMs is less than the number of workers requested, then the caliper manager can split the remaining workers equally across the VMs and display a warning to say it has done this.