Open vponomaryov opened 1 year ago
There's two issues
1) we can't stop the nemesis in flight, if it started it started, and we can only wait for it.
2) we don't housekeep the stress threads, so we can iterate on them and call them and kill specifically, so we iterate on loaders and kill all dockers.
I think we should try and address only the 2nd issue, and stick to each stress thread to be killed only by its code, that can take the different stages of execution into account, and to be aware it was killed.
There's two issues
- we can't stop the nemesis in flight, if it started it started, and we can only wait for it.
- we don't housekeep the stress threads, so we can iterate on them and call them and kill specifically, so we iterate on loaders and kill all dockers.
I think we should try and address only the 2nd issue, and stick to each stress thread to be killed only by its code, that can take the different stages of execution into account, and to be aware it was killed.
Actually, the solution is simple:
And now we have bug just because 4th step is included into the 2nd one.
- stop stress processes keeping docker containers running
how do you do that ? without housekeeping the stress thread objects ?
- stop stress processes keeping docker containers running
how do you do that ? without housekeeping the stress thread objects ?
def __del__(self):
- stop stress processes keeping docker containers running
how do you do that ? without housekeeping the stress thread objects ?
def __del__(self):
but at this stage the object exists, and there a running ssh command docker exec ...
how you are going to stop stress processes keeping docker containers
?
- stop stress processes keeping docker containers running
how do you do that ? without housekeeping the stress thread objects ?
def __del__(self):
but at this stage the object exists, and there a running ssh command
docker exec ...
how you are going tostop stress processes keeping docker containers
?
The docker containers start with idle endless process. Stop of separate stress processes won't cause stop of containers.
- stop stress processes keeping docker containers running
how do you do that ? without housekeeping the stress thread objects ?
def __del__(self):
but at this stage the object exists, and there a running ssh command
docker exec ...
how you are going tostop stress processes keeping docker containers
?The docker containers start with idle endless process. Stop of separate stress processes won't cause stop of containers.
you missing my point, how the teardown is going to know how to stop those processes ? where it's listed all the currently running processes (i.e. loader related ones)
- stop stress processes keeping docker containers running
how do you do that ? without housekeeping the stress thread objects ?
def __del__(self):
but at this stage the object exists, and there a running ssh command
docker exec ...
how you are going tostop stress processes keeping docker containers
?The docker containers start with idle endless process. Stop of separate stress processes won't cause stop of containers.
you missing my point, how the teardown is going to know how to stop those processes ? where it's listed all the currently running processes (i.e. loader related ones)
We do not need to close it explicitly anywhere including the tearDown
.
As I mentioned above, if we use __del__
method then it will be run automatically during garbage collection.
It is enough for our case.
- stop stress processes keeping docker containers running
how do you do that ? without housekeeping the stress thread objects ?
def __del__(self):
but at this stage the object exists, and there a running ssh command
docker exec ...
how you are going tostop stress processes keeping docker containers
?The docker containers start with idle endless process. Stop of separate stress processes won't cause stop of containers.
you missing my point, how the teardown is going to know how to stop those processes ? where it's listed all the currently running processes (i.e. loader related ones)
We do not need to close it explicitly anywhere including the
tearDown
. As I mentioned above, if we use__del__
method then it will be run automatically during garbage collection. It is enough for our case.
so in theory it's enough to just remove the kill_stress part ? what would you do in the __del__
?
Prerequisites
Versions
Logs
Description
The
no_corrupt_repair
nemesis uses loader docker containers to create temporary keyspaces and tables. With the help of the https://github.com/scylladb/scylladb/issues/13322 we can fall into long retry-loop on theget peers info
which times out because of that bug. Then, when scheduled load ends, the Scylla cluster starts responding without timeouts and theget peers info
requests start passing. It leads to continuation of other loop steps.And the problem with it is that the tearDown starts in the middle of the table-creation loop and we may get following error:
It is caused by the racy deletion of the docker containers in the middle of 2 actions which get done by docker loader class:
The SCT code:
And logs:
Steps to Reproduce
Expected behavior: SCT must be smart enough to understand that it killed loaders and not try to use loaders after it.
Actual behavior: SCT allows not-yet stopped nemesis to use loaders which get killed in the tearDown before stopping nemesis.