scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
57 stars 94 forks source link

disrupt_toggle_audit_syslog fails due audit msg not found #7102

Closed soyacz closed 1 month ago

soyacz commented 9 months ago

Issue description

The last check in this nemesis fails due to sdcm.exceptions.AuditLogTestFailure: Audit log is empty while should contain executed DDL (drop table) operation and in sct log we can find "Found 0 audit log rows" while it was audited (visible in db log).

Possibly there's some issue with 'cluster.open_system_log' and not jumping to proper line (based on date) before audited msg - but I can't be sure. Unfortunately, no messages.log were stored to use it for testing locally.

Impact

failing audit nemesis (false error)

How frequently does it reproduce?

unknown, similar was seen in multi-dc scenario (https://github.com/scylladb/scylla-cluster-tests/issues/6619)

Installation details

Kernel Version: 5.15.0-1053-azure Scylla version (or git commit hash): 2023.1.4-20240112.12c616e7f0cf with build-id e7263a4aa92cf866b98cf680bd68d7198c9690c0

Cluster size: 6 nodes (Standard_L8s_v3)

Scylla Nodes used in this run:

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/scylla-images/providers/Microsoft.Compute/images/scylla-2023.1.4-x86_64-2024-01-12T23-58-59 (azure: undefined_region)

Test: longevity-10gb-3h-azure-test Test id: e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b Test name: enterprise-2023.1/longevity/longevity-10gb-3h-azure-test Test config file(s):

Logs and commands - Restore Monitor Stack command: `$ hydra investigate show-monitor e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b) - Show all stored logs command: `$ hydra investigate show-logs e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b` ## Logs: - **db-cluster-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/db-cluster-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/db-cluster-e5f7a9d5.tar.gz) - **sct-runner-events-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/sct-runner-events-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/sct-runner-events-e5f7a9d5.tar.gz) - **sct-e5f7a9d5.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/sct-e5f7a9d5.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/sct-e5f7a9d5.log.tar.gz) - **loader-set-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/loader-set-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/loader-set-e5f7a9d5.tar.gz) - **monitor-set-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/monitor-set-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/monitor-set-e5f7a9d5.tar.gz) - **parallel-timelines-report-e5f7a9d5.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/parallel-timelines-report-e5f7a9d5.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b/20240113_032942/parallel-timelines-report-e5f7a9d5.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/enterprise-2023.1/job/longevity/job/longevity-10gb-3h-azure-test/25/) [Argus](https://argus.scylladb.com/test/495648b9-1123-4547-a33b-eef73457c1db/runs?additionalRuns[]=e5f7a9d5-641d-4c5c-88bc-3e10c50abd0b)
fruch commented 3 months ago

seems to be happening 6 times in the last year

https://70f106c98484448dbc4705050eb3f7e9.us-east-1.aws.found.io:9243/goto/b16ddfd0-3e92-11ef-81c7-3986d18dafd5

fruch commented 2 months ago

we need to check if still happening

soyacz commented 2 months ago

since that time didn't happen...

fruch commented 1 month ago

Still haven't happened since 7July closing this for now