Add ability to change the cluster size

[dorlaor]

It will be great to have a script/cli that changes the cluster size, it needs to be done in stages - there is the level of the VMs (add/remove) and there is the level of the software - decommission or add/install/wait for the cluster to add the node.

[pveentjer]

There is already a set of function where you can execute the following action on scylla nodes:

• clear cluster
• restart cluster
• nodes remove data
• nodes stop
• nodes start

And there is also a SSH/PSSH abstraction so you can execute any command on a node.

So you can run a test like this

cs = CassandraStress(env['loadgenerator_public_ips'], props)
cs.install()
cs.prepare()

cs.upload("stress_example.yaml")

# loading data in cluster
cs.stress(f'user profile=./stress_example.yaml "ops(insert=1)" n=5m -mode native cql3 -rate threads=50 -node {cluster_string}')  

# the actual querying of the system
future = cs.async_stress(f'user profile=./stress_example.yaml "ops(singleclothes=1)" n=40m -log hdrfile=profile.hdr -graph file=profile.html title=query revision=benchmark-0 -mode native cql3 -rate threads=50 -node {cluster_string}')  

sleep(60)

SSH(cluster_public_ips[0], props['cluster_user'], props['ssh_options'])
    .exec("sudo systemctl stop scylla-server")

future.join()

cs.collect_results(iteration.dir)