Closed mikelam14 closed 3 years ago
The saving mechanism in the callback was not implemented with concurrency in mind. What exactly is the problem you're having right now? If you are referring to simultaneous writing to the same path, then currently there's no way to avoid that.
So is there a way to run the training in parallel and save weights after each epoch, maybe with another callback?
Describe the Question
I am using the EvaluateAllMetrics callback, and fit_generator to evaluate at epoch end. The model_save_path works fine if i use 1 worker and use_multiprocessing=False. How do I enable multiprocessing while saving model weights? Thank you
evaluate = mz.callbacks.EvaluateAllMetrics( model, x=pred_x, y=pred_y, batch_size=20, model_save_path='runs/exp')
Describe your attempts