Open vponomaryov opened 2 weeks ago
@soyacz
what do you think, how can we get to a situation we start waiting after the action here ?
First issue is that it discovered late the start line (it appeared at 03:34:39 and detected at 03:34:49), maybe sct was too busy, or message not flushed to disk. I don't know why. I tried analyze the code, but found only saturating cpu issue when waiting for logs: https://github.com/scylladb/scylla-cluster-tests/pull/8743 I don't know if it helps, but I think we should add bit of sleeping before recreating log reader iterator.
First issue is that it discovered late the start line (it appeared at 03:34:39 and detected at 03:34:49), maybe sct was too busy, or message not flushed to disk. I don't know why. I tried analyze the code, but found only saturating cpu issue when waiting for logs: #8743 I don't know if it helps, but I think we should add bit of sleeping before recreating log reader iterator.
maybe we can combine the patterns and wait for both at the same time ?
so at least start using both generator at the same time ?
First issue is that it discovered late the start line (it appeared at 03:34:39 and detected at 03:34:49), maybe sct was too busy, or message not flushed to disk. I don't know why. I tried analyze the code, but found only saturating cpu issue when waiting for logs: #8743 I don't know if it helps, but I think we should add bit of sleeping before recreating log reader iterator.
maybe we can combine the patterns and wait for both at the same time ?
so at least start using both generator at the same time ?
We are starting generators at the same time - before triggering action. That's why this issue is a mystery for me.
Issue description
Running
disrupt_load_and_stream
nemesis we expect appearance ofstart
andend
log messages for theload and stream
operation.In current case in first 2 used nodes logs the expected log lines were correctly found:
node-5:
node-4:
But in case of the next
node-3
theend
log line was not detected and the 5 minutes timeout error was raised: node-3:Steps to Reproduce
disrupt_load_and_stream
nemesisExpected behavior: If a log line exists it must be detected
Actual behavior: Existing log line was not detected
Impact
False negative.
How frequently does it reproduce?
Unknown
Installation details
SCT Version:
master
Scylla version (or git commit hash):master
Logs
f25dd9b8-587a-49cb-bc78-5b8ed3226c68