Open juliayakovlev opened 1 year ago
fruch
commented
1 year ago @juliayakovlev what's the expectation regarding those critical events ? To be suppressed and not shown anywhere ?
The only fix I'm aware of is makeing sure the signal of stopping the test is being sent only once.
juliayakovlev
commented
1 year ago @juliayakovlev what's the expectation regarding those critical events ? To be suppressed and not shown anywhere ?
The only fix I'm aware of is makeing sure the signal of stopping the test is being sent only once.
Yes, we should not send critical c-s event. I remember Dmitiry fixed but did not remember how. Something connected to threads
juliayakovlev
commented
1 year ago sdcm.tester.ClusterTester.stop_resources:
def stop_resources(self): # pylint: disable=no-self-use
self.log.debug('Stopping all resources')
with silence(parent=self, name="Kill Stress Threads"):
self.kill_stress_thread()
# Stopping nemesis, using timeout of 30 minutes, since replace/decommission node can take time
if self.db_cluster:
self.get_nemesis_report(self.db_cluster)
self.stop_nemesis(self.db_cluster)
self.stop_resources_stop_tasks_threads(self.db_cluster)After TearDown starts, we kill stress threads and after that we stop the nemesis.
It this test the stress thread is running from the nemesis. When we kill the stress thread, we get CassandraStressEvent Severity.CRITICAL:
< t:2023-03-30 12:18:38,010 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:INFO > 2023-03-30 12:18:38.008: (CassandraStressEvent Severity.CRITICAL) period_type=end event_id=fd149787-2ddb-450e-9702-7bb5c275c5ae during_nemesis=ReplaceServiceLevelUsingDropDuringLoad duration=24m44s: node=Node longevity-5gb-1h-slareplaceusingdro-loader-node-3a57a28f-eastus-1 [172.174.236.76 | 10.0.0.8] (seed: False)
< t:2023-03-30 12:18:38,010 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:INFO > stress_cmd=cassandra-stress read cl=QUORUM duration=45m -mode cql3 native user=role500_6f8f007e password=rolep500 -rate threads=200 -pop seq=1..1514571 -errors retries=50 -col 'n=FIXED(8) size=FIXED(128)'Maybe we need to change the order: stop nemesis and then stop the stress threads, that still run?
Describe the impact this issue causes to the user.
Describe the frequency with how this issue can be reproduced.
Kernel Version: 5.15.0-1034-azure
Scylla version (or git commit hash): 2023.1.0~rc3-20230321.80de75947b7a with build-id 6e1d6cb6cac9242e7ed7bfd8b07c1fc5998281dc
Cluster size: 3 nodes (Standard_L8s_v3)
Scylla Nodes used in this run:
OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/SCYLLA-IMAGES/providers/Microsoft.Compute/images/scylla-enterprise-2023.1.0-rc3-x86_64-2023-03-23T13-10-32 (azure: eastus)
Test: longevity-5gb-1h-SlaReplaceUsingDropDuringLoad-azure-test
Test id: 3a57a28f-a370-4cd6-a913-b2724b2b3890
Test name: enterprise-2023.1/nemesis/longevity-5gb-1h-SlaReplaceUsingDropDuringLoad-azure-test
Test config file(s):
Test reached its timeout and kill running cassandra-stress thread. In this case critical CassandraStressEvent is not expected. But we got it in the https://jenkins.scylladb.com/job/scylla-master/job/raft/job/longevity-lwt-500G-3d-test/3/
https://argus.scylladb.com/test/3a98b3f1-5534-4e95-a7a9-2f4104f4c851/runs?additionalRuns%5B%5D=062617a6-15f2-46cd-aaed-78b847098fac
AFAIR Dmitry fixed this problem. But now we have it again