search

coiled / benchmarks

BSD 3-Clause "New" or "Revised" License
34 stars 17 forks source link

`test_climatology::test_highlevel_api[execution]` fails at medium scale because event loop is blocked during P2P initalization #1607

Open hendrikmakait opened 1 week ago

hendrikmakait commented 1 week ago

Cluster: https://cloud.coiled.io/clusters/656467/account/dask-benchmarks-gcp/information

This test fails because a worker times out trying to initialize the local P2P state:

Traceback ``` ________________________ test_highlevel_api[execution] _________________________ [gw1] linux -- Python 3.10.15 /home/runner/miniconda3/envs/test/bin/python3.10 > ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/comm/tcp.py:546: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E asyncio.exceptions.CancelledError /opt/coiled/env/lib/python3.10/site-packages/tornado/tcpclient.py:292: CancelledError During handling of the above exception, another exception occurred: > ??? E asyncio.exceptions.CancelledError /opt/coiled/env/lib/python3.10/asyncio/tasks.py:456: CancelledError The above exception was the direct cause of the following exception: > ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/comm/core.py:342: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /opt/coiled/env/lib/python3.10/site-packages/distributed/utils.py:1915: in wait_for ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E asyncio.exceptions.TimeoutError /opt/coiled/env/lib/python3.10/asyncio/tasks.py:458: TimeoutError The above exception was the direct cause of the following exception: > ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_core.py:523: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_rechunk.py:170: in rechunk_transfer ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_worker_plugin.py:348: in add_partition ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_worker_plugin.py:411: in get_or_create_shuffle ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/utils.py:439: in sync ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/utils.py:413: in f ??? /opt/coiled/env/lib/python3.10/site-packages/tornado/gen.py:766: in run ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_worker_plugin.py:145: in get_or_create ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_worker_plugin.py:222: in _refresh ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_worker_plugin.py:[190](https://github.com/coiled/benchmarks/actions/runs/11855888013/job/33041053524#step:9:191): in _fetch ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/core.py:1256: in send_recv_from_rpc ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/core.py:1539: in connect ??? /opt/coiled/env/lib/python3.10/site-packages/distributed/core.py:1429: in _connect ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E OSError: Timed out trying to connect to tls://10.0.32.14:8786 after 30 s /opt/coiled/env/lib/python3.10/site-packages/distributed/comm/core.py:368: OSError The above exception was the direct cause of the following exception: gcs_url = 'gs://coiled-oss-scratch/benchmarks-bot/test-scratch/test_highlevel_api-acf[202](https://github.com/coiled/benchmarks/actions/runs/11855888013/job/33041053524#step:9:203)06ed6d4d4fb8956551c4c29018' scale = 'medium' setup_benchmark = ._ at 0x7f876d6ec0d0> cluster_kwargs = ***'idle_timeout': '1h', 'region': 'us-central1', 'workspace': 'dask-benchmarks-gcp'*** scale_kwargs = ***'large': ***'n_workers': 100***, 'medium': ***'n_workers': 100***, 'small': ***'n_workers': 10*** def test_highlevel_api( gcs_url, scale, setup_benchmark, cluster_kwargs=*** "workspace": "dask-benchmarks-gcp", "region": "us-central1", "idle_timeout": "1h", ***, scale_kwargs=*** "small": ***"n_workers": 10***, "medium": ***"n_workers": 100***, "large": ***"n_workers": 100***, ***, ): with setup_benchmark( **scale_kwargs[scale], **cluster_kwargs ) as benchmark: # noqa: F841 > benchmark( highlevel_api, scale=scale, storage_url=gcs_url, storage_options=***"token": CoiledShippedCredentials()***, ) tests/geospatial/test_climatology.py:60: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/geospatial/conftest.py:[208](https://github.com/coiled/benchmarks/actions/runs/11855888013/job/33041053524#step:9:209): in benchmark_execution func(*args, **kwargs).compute() ../../../miniconda3/envs/test/lib/python3.10/site-packages/dask/base.py:372: in compute (result,) = compute(self, traverse=False, **kwargs) ../../../miniconda3/envs/test/lib/python3.10/site-packages/dask/base.py:660: in compute results = schedule(dsk, keys, **kwargs) /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_rechunk.py:169: in rechunk_transfer ??? /opt/coiled/env/lib/python3.10/contextlib.py:153: in __exit__ ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E RuntimeError: P2P 76f19f5554293784fcd8c9bf1bb07078 failed during transfer phase /opt/coiled/env/lib/python3.10/site-packages/distributed/shuffle/_core.py:531: RuntimeError ```

It looks like this is caused because there is too much load on the scheduler with several rechunks being set up:

Scheduler logs ``` (scheduler) 2024-11-15 12:23:57.625000 tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting (scheduler) 2024-11-15 12:23:59.110000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e29e57a49d7ce9e5063c76fba7d02808 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 45, 0, 0, 0, 0, 45, 0, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:23:59.774000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 45fc967d5a423e195007165c808d57d2 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 45, 6, 0, 0, 0, 45, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:23:59.776000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle a3aa0f43f44758468185f237ef26250e initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 44, 0, 0, 0, 0, 44, 0, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:23:59.778000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 357a14f0213ded2f88bdb1bc686d93ca initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 44, 6, 0, 0, 0, 44, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:00.694000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8795f129f3631547a87e12adc2ff471c initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 46, 0, 0, 0, 0, 46, 0, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:00.736000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle bb160accb4b6cd165996d59e1a3db17b initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 46, 6, 0, 0, 0, 46, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:04.059000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle df7fde64a613c18959b04baa719bdf82 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 40, 0, 0, 0, 0, 40, 0, 0) executed on worker tls://10.0.33.72:41915 (scheduler) 2024-11-15 12:24:04.106000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 1130f7b14341c8fdbfe7b3e120d03971 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 47, 0, 0, 0, 0, 47, 0, 0) executed on worker tls://10.0.32.193:40591 (scheduler) 2024-11-15 12:24:04.159000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 211fde8971926e9e42b3275f9b859f9e initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 47, 6, 0, 0, 0, 47, 21, 0) executed on worker tls://10.0.32.193:40591 (scheduler) 2024-11-15 12:24:04.162000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8857bef460b209472e6102a0e578395a initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 41, 0, 0, 0, 0, 41, 0, 0) executed on worker tls://10.0.33.112:45121 (scheduler) 2024-11-15 12:24:04.175000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8c5d90833dca4d7b244ca18e775c7eaa initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 40, 6, 0, 0, 0, 40, 21, 0) executed on worker tls://10.0.33.112:45121 (scheduler) 2024-11-15 12:24:04.452000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 3dbd653d17fbe745a5ab60140492b65f initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 41, 6, 0, 0, 0, 41, 21, 0) executed on worker tls://10.0.32.193:40591 (scheduler) 2024-11-15 12:24:08.271000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4c3b5c630ae82a23e3827834b253baea initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 42, 6, 0, 0, 0, 42, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:08.295000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 96214bbac5059511973ddbf3ef73b765 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 43, 6, 0, 0, 0, 43, 21, 0) executed on worker tls://10.0.32.52:42499 (scheduler) 2024-11-15 12:24:08.386000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 56d8ec6d5436b325b300a0b80a37be6b initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 36, 6, 0, 0, 0, 36, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:08.453000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle b3ef5d0ef802caa09869924e1cdbc09c initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 43, 0, 0, 0, 0, 43, 0, 0) executed on worker tls://10.0.33.87:39277 (scheduler) 2024-11-15 12:24:08.457000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9f887b32a82f9cbbf8d6f848c2ad68db initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 38, 0, 0, 0, 0, 38, 0, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:08.458000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 145d24eedb0ea2f39f2498bd7c2058cd initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 38, 6, 0, 0, 0, 38, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:08.700000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 43693b2d05f4c0467729e80630fa1d97 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 42, 0, 0, 0, 0, 42, 0, 0) executed on worker tls://10.0.33.79:33383 (scheduler) 2024-11-15 12:24:08.748000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 171c115e0a1c0b252a7c3c0d7df4b993 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 32, 6, 0, 0, 0, 32, 21, 0) executed on worker tls://10.0.33.123:32823 (scheduler) 2024-11-15 12:24:08.759000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6ec6bc7fafa7555569447d4dbab12390 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 39, 6, 0, 0, 0, 39, 21, 0) executed on worker tls://10.0.33.88:41335 (scheduler) 2024-11-15 12:24:08.848000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5997918aad22efb4abe59623fb1a4c7d initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 33, 6, 0, 0, 0, 33, 21, 0) executed on worker tls://10.0.33.190:44123 (scheduler) 2024-11-15 12:24:08.864000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 55ae26b9ca0b747b0b4bd44c95f68f3c initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 34, 6, 0, 0, 0, 34, 21, 0) executed on worker tls://10.0.32.214:36853 (scheduler) 2024-11-15 12:24:08.872000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6ab3efcbb08d6657c00eed84c986c289 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 37, 6, 0, 0, 0, 37, 21, 0) executed on worker tls://10.0.32.201:46521 (scheduler) 2024-11-15 12:24:08.893000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5d014b9988e4360fdbd3a9e282cc48b6 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 33, 0, 0, 0, 0, 33, 0, 0) executed on worker tls://10.0.33.105:43609 (scheduler) 2024-11-15 12:24:08.951000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9d92e31b63758f4fbfe98704d0efb9ab initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 39, 0, 0, 0, 0, 39, 0, 0) executed on worker tls://10.0.33.107:42659 (scheduler) 2024-11-15 12:24:08.998000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 045873a20283a40a8bf17fceaa287b8a initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 32, 0, 0, 0, 0, 32, 0, 0) executed on worker tls://10.0.33.88:41335 (scheduler) 2024-11-15 12:24:09.370000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle ab8d887bbc6b5262a7d6d1111243dd9c initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 36, 0, 0, 0, 0, 36, 0, 0) executed on worker tls://10.0.32.196:35589 (scheduler) 2024-11-15 12:24:09.513000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 954ef81d43375625ea44327d1d66218f initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 37, 0, 0, 0, 0, 37, 0, 0) executed on worker tls://10.0.32.196:35589 (scheduler) 2024-11-15 12:24:09.575000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9437141f024c8ca0f81474260b48166b initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 34, 0, 0, 0, 0, 34, 0, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:10.028000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6210925243de86b04ba6aa6bbc3e367e initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 35, 0, 0, 0, 0, 35, 0, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:10.451000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 98c1f18d7e9329ecea293db87f3e918e initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 56, 0, 0, 0, 0, 56, 2, 0) executed on worker tls://10.0.33.205:41761 (scheduler) 2024-11-15 12:24:12.189000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle f1b9d975605e1ffdb40dec67ecb48814 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 57, 0, 0, 0, 0, 57, 0, 0) executed on worker tls://10.0.32.63:44329 (scheduler) 2024-11-15 12:24:12.191000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle eec900f297bb0a625df4e3774affc1c2 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 35, 6, 0, 0, 0, 35, 21, 0) executed on worker tls://10.0.32.39:46797 (scheduler) 2024-11-15 12:24:12.384000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5059f3b2ab46f406270e6cfcaf1a69eb initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 46, 5, 0, 0, 0, 46, 15, 0) executed on worker tls://10.0.32.24:43519 (scheduler) 2024-11-15 12:24:13.723000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle cc774e00ba5a447eeec11635e4f5b346 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 38, 5, 0, 0, 0, 38, 17, 0) executed on worker tls://10.0.32.207:45821 (scheduler) 2024-11-15 12:24:13.783000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle b4737b6a90ad5076717f2e7a7fe723bd initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 57, 6, 0, 0, 0, 57, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:13.785000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle ae7afdf13b264e742ab29aed1fbd7a06 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 56, 6, 0, 0, 0, 56, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:13.787000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e791cc070737abb3ef55b40a688f58f6 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 58, 0, 0, 0, 0, 58, 0, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:14.069000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 92b884105413ed3b79f9c675464605c3 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 34, 5, 0, 0, 0, 34, 17, 0) executed on worker tls://10.0.33.83:40703 (scheduler) 2024-11-15 12:24:14.307000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle d8f25d2930f81e2459df5e9f9979d9ec initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 40, 5, 0, 0, 0, 40, 15, 0) executed on worker tls://10.0.33.91:40889 (scheduler) 2024-11-15 12:24:14.308000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 86e4eedd40466ad2b131da65b2c7519c initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 63, 5, 0, 0, 0, 63, 16, 0) executed on worker tls://10.0.33.83:40703 (scheduler) 2024-11-15 12:24:14.644000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle a77e82531e98ca1eb855d2998a751f4a initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 50, 0, 0, 0, 0, 50, 2, 0) executed on worker tls://10.0.32.39:46797 (scheduler) 2024-11-15 12:24:14.716000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 293c58eae4e02f5b80c5e70b58e2bc74 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 59, 5, 0, 0, 0, 59, 17, 0) executed on worker tls://10.0.33.71:43743 (scheduler) 2024-11-15 12:24:14.985000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle af9961bfd14a51376037dc28ce567d1c initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 5, 6, 0, 0, 0, 5, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:15.122000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 7f07c1b545692440d20b2c8c7b46804a initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 49, 0, 0, 0, 0, 49, 2, 0) executed on worker tls://10.0.32.207:45821 (scheduler) 2024-11-15 12:24:15.127000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 722311d3fad19ce6abf6cd6af4363298 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 6, 6, 0, 0, 0, 6, 21, 0) executed on worker tls://10.0.33.102:37581 (scheduler) 2024-11-15 12:24:15.185000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle d4155174a716fefc1e6dfabf23b958c4 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 48, 0, 0, 0, 0, 48, 2, 0) executed on worker tls://10.0.32.39:46797 (scheduler) 2024-11-15 12:24:15.190000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 491bde7bd0d2bbff50b15ae901130233 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 50, 6, 0, 0, 0, 50, 21, 0) executed on worker tls://10.0.33.105:43609 (scheduler) 2024-11-15 12:24:15.191000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e06b59debc0a9b4dc685a1f8876b331e initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 63, 0, 0, 0, 0, 63, 2, 0) executed on worker tls://10.0.32.52:42499 (scheduler) 2024-11-15 12:24:15.257000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle fa578a9da7039875e33b7987a16344ff initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 59, 0, 0, 0, 0, 59, 0, 0) executed on worker tls://10.0.32.39:46797 (scheduler) 2024-11-15 12:24:15.371000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle f803e0c966b33a9bebb9927ebb377389 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 55, 6, 0, 0, 0, 55, 21, 0) executed on worker tls://10.0.33.105:43609 (scheduler) 2024-11-15 12:24:15.432000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e2d9b5bfbcac0493e5802ffb54ead446 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 15, 0, 0, 0, 0, 15, 0, 0) executed on worker tls://10.0.32.52:42499 (scheduler) 2024-11-15 12:24:15.437000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e38abf5a8efdcdaf63e5beafb86c6c38 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 60, 6, 0, 0, 0, 60, 21, 0) executed on worker tls://10.0.32.19:43677 (scheduler) 2024-11-15 12:24:15.439000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5434219c6679c12abee94b72864f883f initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 4, 0, 0, 0, 0, 4, 0, 0) executed on worker tls://10.0.32.193:40591 (scheduler) 2024-11-15 12:24:15.440000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 395ecccd3875a43a14be1e96b3b45a18 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 62, 0, 0, 0, 0, 62, 0, 0) executed on worker tls://10.0.32.214:36853 (scheduler) 2024-11-15 12:24:15.503000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9c98696bd53243a8b68654add5ad9c89 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 52, 6, 0, 0, 0, 52, 21, 0) executed on worker tls://10.0.33.114:37667 (scheduler) 2024-11-15 12:24:15.506000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 3390007b88bdf7a60291eb5d344bffee initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 63, 6, 0, 0, 0, 63, 21, 0) executed on worker tls://10.0.33.94:36309 (scheduler) 2024-11-15 12:24:15.508000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8f4ad45405c148c1b8b29c5abdac3b8d initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 10, 0, 0, 0, 0, 10, 0, 0) executed on worker tls://10.0.33.98:45365 (scheduler) 2024-11-15 12:24:15.842000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9453a04afc1e0b531cb9d686691d46a8 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 4, 6, 0, 0, 0, 4, 21, 0) executed on worker tls://10.0.32.196:35589 (scheduler) 2024-11-15 12:24:15.842000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 648607acfd50e4cf48891771393eef5d initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 8, 6, 0, 0, 0, 8, 21, 0) executed on worker tls://10.0.33.100:45089 (scheduler) 2024-11-15 12:24:15.842000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle b6a252233d0a335585f72ad2db801f91 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 59, 6, 0, 0, 0, 59, 21, 0) executed on worker tls://10.0.33.68:42835 (scheduler) 2024-11-15 12:24:15.842000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 48896ca742812ea0811b5019a8aeeb55 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 52, 0, 0, 0, 0, 52, 0, 0) executed on worker tls://10.0.32.19:43677 (scheduler) 2024-11-15 12:24:15.842000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 744dc3b358f4fbea09f9ce20d638bc3a initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 51, 6, 0, 0, 0, 51, 21, 0) executed on worker tls://10.0.33.115:34281 (scheduler) 2024-11-15 12:24:15.842000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle f5f01e2bcb7d5a388752dde0cc2b69a5 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 49, 6, 0, 0, 0, 49, 21, 0) executed on worker tls://10.0.33.119:35221 (scheduler) 2024-11-15 12:24:16.177000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 08d30c5098f41d0ba7f6ebc0e7c00908 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 53, 6, 0, 0, 0, 53, 21, 0) executed on worker tls://10.0.33.198:39349 (scheduler) 2024-11-15 12:24:16.178000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 2eaf46dab7566c56ad06ccf774ca898a initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 54, 0, 0, 0, 0, 54, 0, 0) executed on worker tls://10.0.33.94:36309 (scheduler) 2024-11-15 12:24:16.178000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 349fdeba97b519430070d446cb5eac56 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 55, 0, 0, 0, 0, 55, 2, 0) executed on worker tls://10.0.33.116:36687 (scheduler) 2024-11-15 12:24:16.178000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 046a19188ff34656cac12d1912ee8f9a initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 12, 6, 0, 0, 0, 12, 21, 0) executed on worker tls://10.0.33.115:34281 (scheduler) 2024-11-15 12:24:16.178000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 92d3df430bfb7c4ff5e18667f7b09e63 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 61, 6, 0, 0, 0, 61, 21, 0) executed on worker tls://10.0.33.68:42835 (scheduler) 2024-11-15 12:24:16.178000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 90c198f3e3d5ea55be48fcad07064d80 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) executed on worker tls://10.0.33.100:45089 (scheduler) 2024-11-15 12:24:16.515000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e12a846e098f2d93d9dc6a50dd607d00 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 1, 0, 0, 0, 0, 1, 0, 0) executed on worker tls://10.0.32.19:43677 (scheduler) 2024-11-15 12:24:16.515000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8c2a41d12ae847b494174b29627a5df1 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 47, 5, 0, 0, 0, 47, 17, 0) executed on worker tls://10.0.33.66:38767 (scheduler) 2024-11-15 12:24:16.515000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 653626c1fd8148808cf121b2eb9e3fae initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 61, 0, 0, 0, 0, 61, 0, 0) executed on worker tls://10.0.47.194:34447 (scheduler) 2024-11-15 12:24:16.515000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4eec182a2a88e237bf8246baaa3336cc initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 61, 5, 0, 0, 0, 61, 16, 0) executed on worker tls://10.0.32.24:43519 (scheduler) 2024-11-15 12:24:16.515000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8dfc8fa13510e1ae4d4ec1ffed52e1c1 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 5, 0, 0, 0, 0, 5, 0, 0) executed on worker tls://10.0.33.98:45365 (scheduler) 2024-11-15 12:24:16.515000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle f72db1200009854b416ef870cf092337 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 11, 6, 0, 0, 0, 11, 21, 0) executed on worker tls://10.0.32.19:43677 (scheduler) 2024-11-15 12:24:16.850000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6281f0a9e03e5a4f9a4701f895a5f936 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 53, 0, 0, 0, 0, 53, 0, 0) executed on worker tls://10.0.33.67:34837 (scheduler) 2024-11-15 12:24:16.850000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e8a24a85f6049a5ddf6e7ad82ab49a46 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 14, 0, 0, 0, 0, 14, 0, 0) executed on worker tls://10.0.33.76:34579 (scheduler) 2024-11-15 12:24:16.850000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 363f2c843952385d19f64b4c4b0c6658 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 8, 0, 0, 0, 0, 8, 0, 0) executed on worker tls://10.0.33.67:34837 (scheduler) 2024-11-15 12:24:16.850000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 58087b95bb9d916d2a2a3d2d564a3a11 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 62, 6, 0, 0, 0, 62, 21, 0) executed on worker tls://10.0.33.110:35611 (scheduler) 2024-11-15 12:24:16.850000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5bdebd6b6a2e4a7686e6dcd6b6e83e40 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 0, 6, 0, 0, 0, 0, 21, 0) executed on worker tls://10.0.33.70:44803 (scheduler) 2024-11-15 12:24:17.187000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4393b5b1e530d0826acc8457dd6e5c44 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 51, 0, 0, 0, 0, 51, 0, 0) executed on worker tls://10.0.33.201:41649 (scheduler) 2024-11-15 12:24:17.188000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 2efbeaac6cae0d5d3e341bfa0b4cd608 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 60, 0, 0, 0, 0, 60, 0, 0) executed on worker tls://10.0.33.84:37291 (scheduler) 2024-11-15 12:24:17.188000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 33152a47bced48405f3408b302982210 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 58, 6, 0, 0, 0, 58, 21, 0) executed on worker tls://10.0.33.77:44071 (scheduler) 2024-11-15 12:24:17.188000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 27053b0f2cbb19da34bdaaafcb472d4d initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 7, 6, 0, 0, 0, 7, 21, 0) executed on worker tls://10.0.32.30:33985 (scheduler) 2024-11-15 12:24:17.188000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9b6dbb4dc5b34a74f0cd6526dd15f79b initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 12, 0, 0, 0, 0, 12, 0, 0) executed on worker tls://10.0.33.108:37053 (scheduler) 2024-11-15 12:24:17.188000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e86d3fae3726f67f862de33a7a23337a initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 39, 5, 0, 0, 0, 39, 17, 0) executed on worker tls://10.0.33.110:35611 (scheduler) 2024-11-15 12:25:11.907000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 2532d0371dd576844c14b64e64bcb4a9 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 13, 6, 0, 0, 0, 13, 21, 0) executed on worker tls://10.0.33.121:35045 (scheduler) 2024-11-15 12:25:11.908000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle f6f87ef70dea1328b6512e92ad046526 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 15, 6, 0, 0, 0, 15, 21, 0) executed on worker tls://10.0.33.121:35045 (scheduler) 2024-11-15 12:25:11.909000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8746f35e116fc6df698ff876813909ba initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 1, 6, 0, 0, 0, 1, 21, 0) executed on worker tls://10.0.33.121:35045 (scheduler) 2024-11-15 12:25:11.911000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 276775d2d7aaec3d3949dea194c90f61 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 48, 6, 0, 0, 0, 48, 21, 0) executed on worker tls://10.0.33.121:35045 (scheduler) 2024-11-15 12:25:11.974000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 31f138693f2b8fa63efbc484b09885bc initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 6, 0, 0, 0, 0, 6, 0, 0) executed on worker tls://10.0.33.67:34837 (scheduler) 2024-11-15 12:25:11.976000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 2f1e2e42eecc3e33aaac25982a1b3e31 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 10, 6, 0, 0, 0, 10, 21, 0) executed on worker tls://10.0.33.70:44803 (scheduler) 2024-11-15 12:25:12.000000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 02ec21fab98ded0c4a22340444228a14 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 47, 1, 0, 0, 0, 47, 3, 0) executed on worker tls://10.0.32.57:44939 (scheduler) 2024-11-15 12:25:12.094000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle e52b310762ef37628383c192a12ae266 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 13, 0, 0, 0, 0, 13, 0, 0) executed on worker tls://10.0.33.70:44803 (scheduler) 2024-11-15 12:25:12.130000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 7f146477a37f443c326417a35797f3cd initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 9, 0, 0, 0, 0, 9, 0, 0) executed on worker tls://10.0.33.106:42101 (scheduler) 2024-11-15 12:25:12.132000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 294fdef2a47a656fb3a0fd8a09d69f61 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 54, 6, 0, 0, 0, 54, 21, 0) executed on worker tls://10.0.33.106:42101 (scheduler) 2024-11-15 12:25:12.134000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 98c27d3c70fb0aad512bc1b712e6515e initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 11, 0, 0, 0, 0, 11, 0, 0) executed on worker tls://10.0.33.118:46859 (scheduler) 2024-11-15 12:25:12.469000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0acbdbb6745f47c3b2e1c3a4b0de218d initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 14, 6, 0, 0, 0, 14, 21, 0) executed on worker tls://10.0.33.118:46859 (scheduler) 2024-11-15 12:25:12.469000 distributed.core - INFO - Event loop was unresponsive in Scheduler for 56.32s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. (scheduler) 2024-11-15 12:25:12.531000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle c8665cce593615cba052ae468a2d0bb4 initialized by task ('rechunk-transfer-c81429b79c85f3b682f04c128fa8fddc', 0, 0, 7, 0, 0, 0, 0, 7, 0, 0) executed on worker tls://10.0.33.109:39251 (scheduler) 2024-11-15 12:25:26.230000 distributed.shuffle._scheduler_plugin - WARNING - Shuffle 357a14f0213ded2f88bdb1bc686d93ca deactivated due to stimulus 'task-erred-1731673486.0191505' ``` <