CI Failure (Set changed size during iteration) in `EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures`

[vbotbuildovich]

https://buildkite.com/redpanda/redpanda/builds/48605

Module: rptest.tests.e2e_shadow_indexing_test
Class: EndToEndShadowIndexingTestWithDisruptions
Method: test_write_with_node_failures
Arguments: {
    "cloud_storage_type": 2
}

test_id:    EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures
status:     FAIL
run time:   47.491 seconds

RuntimeError('Set changed size during iteration')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 103, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/e2e_shadow_indexing_test.py", line 894, in test_write_with_node_failures
    wait_for_local_storage_truncate(redpanda=self.redpanda,
  File "/root/tests/rptest/util.py", line 287, in wait_for_local_storage_truncate
    wait_until(
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 53, in wait_until
    raise e
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 44, in wait_until
    if condition():
  File "/root/tests/rptest/util.py", line 241, in is_truncated
    storage = redpanda.storage(sizes=True, scan_cache=False)
  File "/root/tests/rptest/services/redpanda.py", line 4046, in storage
    self.for_nodes(nodes, compute_node_storage)
  File "/root/tests/rptest/services/redpanda.py", line 1391, in for_nodes
    return list(executor.map(cb, nodes))
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 610, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 610, in <listcomp>
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
RuntimeError: Set changed size during iteration
Test requested 5 nodes, used only 4```

JIRA Link: [CORE-2759](https://redpandadata.atlassian.net/browse/CORE-2759)

[CORE-2759]: https://redpandadata.atlassian.net/browse/CORE-2759?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/48769

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/51716

[piyushredpanda]

Closing older-bot-filed CI issues as we transition to a more reliable system.