Is your feature request related to a problem? Please describe
CUDA_VISIBLE_DEVICES is frequently used for cuda device management, and currently, this environment variable can be assigned when setting up actor pools, resulting in fixed cuda device allocation.
Nonetheless, there are situations where dynamic cuda device allocation is beneficial. For instance, when performing inference on models of varying sizes, the required number of cuda devices may differ. To optimize cuda device utilization effectively, it is advantageous to determine CUDA_VISIBLE_DEVICES at runtime based on the model size.
Is your feature request related to a problem? Please describe
CUDA_VISIBLE_DEVICESis frequently used for cuda device management, and currently, this environment variable can be assigned when setting up actor pools, resulting in fixed cuda device allocation.Nonetheless, there are situations where dynamic cuda device allocation is beneficial. For instance, when performing inference on models of varying sizes, the required number of cuda devices may differ. To optimize cuda device utilization effectively, it is advantageous to determine
CUDA_VISIBLE_DEVICESat runtime based on the model size.Describe the solution you'd like
Support adding and removing subpools.