search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
57 stars 94 forks source link

Longevity-lwt-3h tried only one nemesis and didn't progress #4261

Closed roydahan closed 5 months ago

roydahan commented 2 years ago

In job: "scylla-master/longevity/longevity-lwt-3h-test#312: 2021-12-22 09:35:39" the test chose the first nemesis (LdapConnectionToggle) and started it at 10:18:44,357. The nemesis should have been skipped because it's not an enterprise release, but it didn't progress at all and the next nemesis never started.

From sct.log:

< t:2021-12-22 10:17:17,377 f:nemesis.py      l:1190 c:sdcm.nemesis         p:INFO  > sdcm.nemesis.SisyphusMonkey: >>>>>>>>>>>>>Started random_disrupt_method ldap_connection_toggle
< t:2021-12-22 10:17:17,378 f:nemesis.py      l:295  c:sdcm.nemesis         p:INFO  > sdcm.nemesis.SisyphusMonkey: Current Target: Node longevity-lwt-3h-master-db-node-3842ca8a-4 [13.48.13.143 | 10.0.0.224] (seed: False) with running nemesis: None
< t:2021-12-22 10:18:44,357 f:nemesis.py      l:3050 c:sdcm.nemesis         p:DEBUG > sdcm.nemesis.SisyphusMonkey: Start disruption at `2021-12-22 10:18:44.357972`
< t:2021-12-22 10:18:44,358 f:nemesis.py      l:777  c:sdcm.nemesis         p:DEBUG > sdcm.nemesis.SisyphusMonkey: Set current_disruption -> LdapConnectionToggle Node longevity-lwt-3h-master-db-node-3842ca8a-4 [13.48.13.143 | 10.0.0.224] (seed: False)

If this issue repeats in next runs, it should be handled with high severity. If it was one time failure, we can ignore it for now and just track it.

roydahan commented 2 years ago

@fgelcer since you added this nemesis, you won the issue. I re-ran the job and it's still stuck on this nemesis, please look into it and send a fix: https://jenkins.scylladb.com/job/scylla-master/job/longevity/job/longevity-lwt-3h-test/313/

fgelcer commented 2 years ago

@roydahan , LDAP shouldn't run for OSS (support only for enterprise).

fruch commented 2 years ago

happens again in https://jenkins.scylladb.com/job/scylla-master/job/longevity/job/longevity-lwt-3h-test/322

roydahan commented 2 years ago

@fgelcer please check it.

fgelcer commented 2 years ago

@fgelcer please check it.

apparently, it seems to be doing the right thing:

2022-01-16 05:22:25.606: (DisruptionEvent Severity.NORMAL) period_type=begin event_id=f8fcaf70-837f-4ed2-986d-5144779cc45e: nemesis_name=LdapConnectionToggle target_node=Node longevity-lwt-3h-master-db-node-1549e9bd-2 [54.73.38.75 | 10.0.3.147] (seed: False)
2022-01-16 05:22:25.762: (DisruptionEvent Severity.NORMAL) period_type=end event_id=f8fcaf70-837f-4ed2-986d-5144779cc45e duration=0s: nemesis_name=LdapConnectionToggle target_node=Node longevity-lwt-3h-master-db-node-1549e9bd-2 [54.73.38.75 | 10.0.3.147] (seed: False) skipped skip_reason=Cluster is not configured to run with LDAP authorization, hence skipping

exactly like in this line

now, might be something with Sisyphus monkey? the disrupt_ldap_connection_toggle is the 1st nemesis in the list, but after it was skipped, why wasn't the next one (disrupt_mgmt_repair_cli) called?

roydahan commented 2 years ago

Check the sct.log. If it reaches back to Sisyphus it should pop the next one. We have other examples where nemesis skipped with Sisyphus.

fgelcer commented 2 years ago

grepping sct.log... the 1st nemesis wasn't supposed to be the LDAP one...

< t:2022-01-16 04:09:23,256 f:sct_config.py   l:1524 c:sdcm.sct_config      p:INFO  > nemesis_class_name: SisyphusMonkey
< t:2022-01-16 04:32:50,942 f:nemesis.py      l:1252 c:sdcm.nemesis         p:DEBUG > sdcm.nemesis.SisyphusMonkey: This is the list of callable disruptions [<bound method Nemesis.disrupt_abort_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_drop_column of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_remove_dc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_corrupt_then_scrub of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_decommission_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_10_full_partitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_by_rows_range of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_rebuild of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_disable_enable_ldap_authorization of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_new_rack of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hot_reloading_internode_certificate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_kill_scylla of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_load_and_stream of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_memory_stress of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup_specific_keyspaces of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_repair_cli of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_modify_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_multiple_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_block of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_random_interruptions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_node_exporter of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_thrift of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_start_stop_interface of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_no_corrupt_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_cleanup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_drain of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_enospc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_refresh of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_seed_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rebuild_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_remove_node_then_add_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_repair_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_resetlocalschema of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_restart_with_resharding of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_config_change_internode_compression of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_restart_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_cdcstressor_tool of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_unique_sequence of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_show_toppartitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_snapshot_operations of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_soft_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_cleanup_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_scrub_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_validation_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_wait_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_switch_between_password_authenticator_and_saslauthd_authenticator_and_back of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_decommission_add_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_cdc_feature_properties_on_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_table_ics of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate_large_partition of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_abort_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_drop_column of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_remove_dc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_corrupt_then_scrub of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_decommission_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_10_full_partitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_by_rows_range of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_rebuild of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_disable_enable_ldap_authorization of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_new_rack of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hot_reloading_internode_certificate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_kill_scylla of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_load_and_stream of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_memory_stress of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup_specific_keyspaces of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_repair_cli of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_modify_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_multiple_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_block of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_random_interruptions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_node_exporter of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_thrift of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_start_stop_interface of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_no_corrupt_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_cleanup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_drain of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_enospc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_refresh of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_seed_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rebuild_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_remove_node_then_add_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_repair_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_resetlocalschema of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_restart_with_resharding of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_config_change_internode_compression of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_restart_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_cdcstressor_tool of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_unique_sequence of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_show_toppartitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_snapshot_operations of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_soft_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_cleanup_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_scrub_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_validation_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_wait_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_switch_between_password_authenticator_and_saslauthd_authenticator_and_back of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_decommission_add_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_cdc_feature_properties_on_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_table_ics of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate_large_partition of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_abort_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_drop_column of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_remove_dc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_corrupt_then_scrub of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_decommission_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_10_full_partitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_by_rows_range of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_rebuild of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_disable_enable_ldap_authorization of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_new_rack of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hot_reloading_internode_certificate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_kill_scylla of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_load_and_stream of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_memory_stress of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup_specific_keyspaces of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_repair_cli of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_modify_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_multiple_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_block of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_random_interruptions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_node_exporter of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_thrift of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_start_stop_interface of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_no_corrupt_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_cleanup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_drain of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_enospc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_refresh of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_seed_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rebuild_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_remove_node_then_add_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_repair_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_resetlocalschema of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_restart_with_resharding of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_config_change_internode_compression of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_restart_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_cdcstressor_tool of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_unique_sequence of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_show_toppartitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_snapshot_operations of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_soft_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_cleanup_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_scrub_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_validation_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_wait_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_switch_between_password_authenticator_and_saslauthd_authenticator_and_back of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_decommission_add_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_cdc_feature_properties_on_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_table_ics of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate_large_partition of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_abort_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_drop_column of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_remove_dc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_corrupt_then_scrub of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_decommission_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_10_full_partitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_by_rows_range of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_rebuild of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_disable_enable_ldap_authorization of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_new_rack of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hot_reloading_internode_certificate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_kill_scylla of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_load_and_stream of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_memory_stress of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup_specific_keyspaces of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_repair_cli of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_modify_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_multiple_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_block of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_random_interruptions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_node_exporter of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_thrift of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_start_stop_interface of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_no_corrupt_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_cleanup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_drain of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_enospc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_refresh of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_seed_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rebuild_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_remove_node_then_add_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_repair_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_resetlocalschema of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_restart_with_resharding of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_config_change_internode_compression of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_restart_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_cdcstressor_tool of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_unique_sequence of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_show_toppartitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_snapshot_operations of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_soft_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_cleanup_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_scrub_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_validation_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_wait_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_switch_between_password_authenticator_and_saslauthd_authenticator_and_back of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_decommission_add_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_cdc_feature_properties_on_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_table_ics of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate_large_partition of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_abort_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_drop_column of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_remove_dc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_corrupt_then_scrub of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_decommission_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_10_full_partitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_by_rows_range of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_rebuild of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_disable_enable_ldap_authorization of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_new_rack of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hot_reloading_internode_certificate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_kill_scylla of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_load_and_stream of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_memory_stress of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup_specific_keyspaces of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_repair_cli of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_modify_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_multiple_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_block of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_random_interruptions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_node_exporter of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_thrift of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_start_stop_interface of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_no_corrupt_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_cleanup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_drain of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_enospc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_refresh of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_seed_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rebuild_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_remove_node_then_add_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_repair_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_resetlocalschema of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_restart_with_resharding of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_config_change_internode_compression of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_restart_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_cdcstressor_tool of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_unique_sequence of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_show_toppartitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_snapshot_operations of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_soft_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_cleanup_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_scrub_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_validation_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_wait_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_switch_between_password_authenticator_and_saslauthd_authenticator_and_back of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_decommission_add_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_cdc_feature_properties_on_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_table_ics of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate_large_partition of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_abort_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_drop_column of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_add_remove_dc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_corrupt_then_scrub of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_decommission_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_10_full_partitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_delete_by_rows_range of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_rebuild of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_disable_enable_ldap_authorization of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_grow_shrink_new_rack of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_hot_reloading_internode_certificate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_kill_scylla of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_load_and_stream of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_memory_stress of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_backup_specific_keyspaces of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_repair_cli of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_modify_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_multiple_hard_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_block of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_random_interruptions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_node_exporter of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_thrift of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_start_stop_interface of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_no_corrupt_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_cleanup of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_drain of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_enospc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_refresh of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_seed_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rebuild_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_remove_node_then_add_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_repair_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_resetlocalschema of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_restart_with_resharding of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_config_change_internode_compression of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_rolling_restart_cluster of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_cdcstressor_tool of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_run_unique_sequence of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_show_toppartitions of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_snapshot_operations of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_soft_reboot_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_cleanup_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_major_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_scrub_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_start_stop_validation_compaction of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_stop_wait_start_scylla_server of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_switch_between_password_authenticator_and_saslauthd_authenticator_and_back of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_and_replace_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_terminate_decommission_add_node_kubernetes of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_cdc_feature_properties_on_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_table_ics of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate_large_partition of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>]

but grepping the events.log for disruption events we have:

$ grep DisruptionEvent events-1549e9bd.log 
2022-01-16 05:22:25.606: (DisruptionEvent Severity.NORMAL) period_type=begin event_id=f8fcaf70-837f-4ed2-986d-5144779cc45e: nemesis_name=LdapConnectionToggle target_node=Node longevity-lwt-3h-master-db-node-1549e9bd-2 [54.73.38.75 | 10.0.3.147] (seed: False)
2022-01-16 05:22:25.762: (DisruptionEvent Severity.NORMAL) period_type=end event_id=f8fcaf70-837f-4ed2-986d-5144779cc45e duration=0s: nemesis_name=LdapConnectionToggle target_node=Node longevity-lwt-3h-master-db-node-1549e9bd-2 [54.73.38.75 | 10.0.3.147] (seed: False) skipped skip_reason=Cluster is not configured to run with LDAP authorization, hence skipping
fgelcer commented 2 years ago

@roydahan , seems like a bug in the nemesis thread when selecting the nemesis from the pre-created list

roydahan commented 2 years ago

nemesis stack AFTER SHUFFLE is [<bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_repair_cli of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_cdc_feature_properties_on_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_drain of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate_large_partition of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_repair_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_rebuild of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_rebuild of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_network_reject_node_exporter of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_load_and_stream of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_destroy_data_then_repair of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_enospc of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_decommission of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>,

roydahan commented 2 years ago

sdcm.nemesis.SisyphusMonkey: Selecting the next nemesis out of stack [<bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_mgmt_repair_cli of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_ldap_connection_toggle of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_toggle_cdc_feature_properties_on_table of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_nodetool_drain of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_truncate_large_partition of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>, <bound method Nemesis.disrupt_repair_streaming_err of <sdcm.nemesis.SisyphusMonkey object at 0x7fd6d6bd2a10>>,

roydahan commented 2 years ago

@fgelcer the ball back to you.

roydahan commented 2 years ago

Is it still the case? Please handle this in the current sprint if it's still happening.

fgelcer commented 2 years ago

on 5.0 it didn't happen in the last run

last one on master is failing, but not sure it is the same as in this issue

github-actions[bot] commented 5 months ago

This issue is stale because it has been open 2 years with no activity. Remove stale label or comment or this will be closed in 2 days.

github-actions[bot] commented 5 months ago

This issue was closed because it has been stalled for 2 days with no activity.