Open tgaddair opened 2 years ago
cc @clarkzinzow
Hey @tgaddair a workaround would be to use max_concurrent_trials
argument in tune.run
. This ensures that no more than N trials will be scheduled at a time. If you set it so that some resources are unclaimed by trials, Ray Data workers will be able to run. In case of a dynamic resource allocation, the function used for resources will need to be modified to keep some CPUs free (should be trivial).
We'll be looking into making a proper fix for this.
Seems there is a deadlock that arises when using Ray Tune with dynamic resource allocation + RayDatasets. When we set
cache_format = 'parquet'
, everything works fine, but when we use the new defaultcache_format = 'ray'
, trials will hang, presumably because RayDatasets are locking out some of the resources needed by dynamic allocation.Even if we bump up the number of resources in the cluster, we end up in the same place: