[x] separate encoding and tuning
[x] modify PredictWrapper to get the data already loaded by dataloader
Both modifications will ensure that multiprocessing.pool (used during the loading and encoding of the data) will not compete for the threading sources with ray tune (either during training or predicting the performance in the objective function).
Hope this will help in the deadlock problem. If not, at least it will avoid loading and encoding the data multiple times...
[x] separate encoding and tuning [x] modify PredictWrapper to get the data already loaded by dataloader
Both modifications will ensure that multiprocessing.pool (used during the loading and encoding of the data) will not compete for the threading sources with ray tune (either during training or predicting the performance in the objective function).
Hope this will help in the deadlock problem. If not, at least it will avoid loading and encoding the data multiple times...
Also, probably multiprocessing is also used by other packages (DataLoader, etc) Related issue: https://github.com/ray-project/ray/issues/28197
Added another modification: [x] pass the data through tune.with_parameters to the workers. In this way, the data is passed as a pointer.