ignore_mutation_write_errors context manager does not decrease severity

[soyacz]

ignore_mutation_write_errors context manager does not decrease severity of events like:

2022-05-22 09:47:09.531 <2022-05-22 09:47:09.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=393debaf-3a91-4611-8d8d-105c6be8ad6c: type=DATABASE_ERROR regex=Exception  line_number=4084 node=longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-2
2022-05-22T09:47:09+00:00 longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-2 !     ERR |  [shard 2] storage_proxy - exception during mutation write to 10.0.1.26: std::runtime_error (Column definition lwt_indicator does not match any column in the query selection)
022-05-22T09:47:19+00:00 longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-1 !     ERR |  [shard 2] storage_proxy - exception during mutation write to 10.0.0.32: std::runtime_error (Column definition lwt_indicator does not match any column in the query selection)

although in code we can see that it should be logged as warning:

@contextmanager
def ignore_mutation_write_errors():
    with ExitStack() as stack:
        stack.enter_context(EventsSeverityChangerFilter(
            new_severity=Severity.WARNING,
            event_class=LogEvent,
            regex=r".*mutation_write_",
            extra_time_to_expiration=30
        ))
[...]

This happened in LWT test where we change schema. Details of the test:

Installation details

Kernel Version: 5.13.0-1022-aws Scylla version (or git commit hash): 5.0~rc5-20220515.9da666e77 with build-id 5992da65161c733b3f85b8c93bbbb5151d4d321c Cluster size: 4 nodes (i3.2xlarge)

Scylla Nodes used in this run:

• longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-7 (54.246.66.64 | 10.0.0.190) (shards: 8)
• longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-6 (34.240.135.251 | 10.0.3.231) (shards: 8)
• longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-5 (52.212.125.236 | 10.0.3.138) (shards: 8)
• longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-4 (54.154.29.205 | 10.0.0.33) (shards: 8)
• longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-3 (54.246.73.96 | 10.0.3.28) (shards: 8)
• longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-2 (34.245.75.205 | 10.0.1.26) (shards: 8)
• longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-1 (54.246.61.251 | 10.0.0.32) (shards: 8)

OS / Image: ami-028dc71f18948aff6 (aws: eu-west-1)

Test: longevity-lwt-parallel-schema-changes-with-disruptive-24h-test Test id: bdf8bc67-60ac-43d5-bbf4-a5e0a316086d Test name: scylla-5.0/longevity/longevity-lwt-parallel-schema-changes-with-disruptive-24h-test Test config file(s):

• longevity-lwt-parallel-24h.yaml

Issue description

>>>>>>> Your description here... <<<<<<<

• Restore Monitor Stack command: $ hydra investigate show-monitor bdf8bc67-60ac-43d5-bbf4-a5e0a316086d
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs bdf8bc67-60ac-43d5-bbf4-a5e0a316086d

Logs:

• db-cluster-bdf8bc67.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/db-cluster-bdf8bc67.tar.gz
• monitor-set-bdf8bc67.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/monitor-set-bdf8bc67.tar.gz
• loader-set-bdf8bc67.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/loader-set-bdf8bc67.tar.gz
• normal-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/normal-bdf8bc67.log.tar.gz
• summary-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/summary-bdf8bc67.log.tar.gz
• events-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/events-bdf8bc67.log.tar.gz
• output-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/output-bdf8bc67.log.tar.gz
• debug-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/debug-bdf8bc67.log.tar.gz
• sct-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/sct-bdf8bc67.log.tar.gz
• error-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/error-bdf8bc67.log.tar.gz
• critical-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/critical-bdf8bc67.log.tar.gz
• raw_events-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/raw_events-bdf8bc67.log.tar.gz
• warning-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/warning-bdf8bc67.log.tar.gz
• email_data-bdf8bc67.json.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/email_data-bdf8bc67.json.tar.gz
• argus-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/argus-bdf8bc67.log.tar.gz
• left_processes-bdf8bc67.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bdf8bc67-60ac-43d5-bbf4-a5e0a316086d/20220523_063720/left_processes-bdf8bc67.log.tar.gz

Jenkins job URL

[fruch]

the regex in the filter doesn't match the lines you are seeing, if you think they should part of that, change the regex to make sure it cover them as well (after you verified that this isn't surfacing a real issue here)

[soyacz]

Yes, I saw it, anyway I created an issue to discuss it if it's still relevant. I think @juliayakovlev was writing these tests. @juliayakovlev what's your opinion?

[juliayakovlev]

@soyacz

Event in your example search for Exception, not ".*mutation_write_":

2022-05-22 09:47:09.531 <2022-05-22 09:47:09.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=393debaf-3a91-4611-8d8d-105c6be8ad6c: type=DATABASE_ERROR regex=Exception  line_number=4084 node=longevity-lwt-parallel-24h-5-0-db-node-bdf8bc67-2

Also the error is mutation write and not mutation_write_. It's other error, not that should be decreased.

So it's not connected to the contex manager.

Please, do not remove mutation_write_ regex

[soyacz]

Thanks @juliayakovlev. I'm closing the issue.