This may be a network issue with NVIDIA security groups or an issue with dask-cloud provider, which is reportedly needs more resources. Found during 23.12 deployment testing. cc @aravenel @jacobtomlinson
Proposed solutions:
Test with a more permission security group, closer to our customer's or average user
If the issue persists after testing one, allocate resources to fix any bugs in dask cloud provider.
Deploying an ec2-mnmg cluster setup does not properly connect the workers that were created (and can be shut down) properly.
https://docs.rapids.ai/deployment/nightly/cloud/aws/ec2-multi/#cluster-setup
This may be a network issue with NVIDIA security groups or an issue with dask-cloud provider, which is reportedly needs more resources. Found during 23.12 deployment testing. cc @aravenel @jacobtomlinson
Proposed solutions: