fruch
commented
3 months ago seems to be happening 6 times in the last year
Closed soyacz closed 1 month ago
fruch
commented
3 months ago seems to be happening 6 times in the last year
fruch
commented
2 months ago we need to check if still happening
soyacz
commented
2 months ago since that time didn't happen...
fruch
commented
1 month ago Still haven't happened since 7July closing this for now
Issue description
The last check in this nemesis fails due to
sdcm.exceptions.AuditLogTestFailure: Audit log is empty while should contain executed DDL (drop table) operationand in sct log we can find "Found 0 audit log rows" while it was audited (visible in db log).Possibly there's some issue with 'cluster.open_system_log' and not jumping to proper line (based on date) before audited msg - but I can't be sure. Unfortunately, no messages.log were stored to use it for testing locally.
Impact
failing audit nemesis (false error)
How frequently does it reproduce?
unknown, similar was seen in multi-dc scenario (https://github.com/scylladb/scylla-cluster-tests/issues/6619)
Installation details
Kernel Version: 5.15.0-1053-azure Scylla version (or git commit hash):
2023.1.4-20240112.12c616e7f0cfwith build-ide7263a4aa92cf866b98cf680bd68d7198c9690c0Cluster size: 6 nodes (Standard_L8s_v3)
Scylla Nodes used in this run:
OS / Image:
/subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/scylla-images/providers/Microsoft.Compute/images/scylla-2023.1.4-x86_64-2024-01-12T23-58-59(azure: undefined_region)Test:
longevity-10gb-3h-azure-testTest id:e5f7a9d5-641d-4c5c-88bc-3e10c50abd0bTest name:enterprise-2023.1/longevity/longevity-10gb-3h-azure-testTest config file(s):Logs and commands
- Restore Monitor Stack command: `$ hydra investigate show-monitor e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b) - Show all stored logs command: `$ hydra investigate show-logs e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b` ## Logs: - **db-cluster-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/db-cluster-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/db-cluster-e5f7a9d5.tar.gz) - **sct-runner-events-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/sct-runner-events-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/sct-runner-events-e5f7a9d5.tar.gz) - **sct-e5f7a9d5.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/sct-e5f7a9d5.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/sct-e5f7a9d5.log.tar.gz) - **loader-set-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/loader-set-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/loader-set-e5f7a9d5.tar.gz) - **monitor-set-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/monitor-set-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/monitor-set-e5f7a9d5.tar.gz) - **parallel-timelines-report-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/parallel-timelines-report-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/parallel-timelines-report-e5f7a9d5.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/enterprise-2023.1/job/longevity/job/longevity-10gb-3h-azure-test/25/) [Argus](https://argus.scylladb.com/test/495648b9-1123-4547-a33b-eef73457c1db/runs?additionalRuns[]=e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b)