NTMC-Community / MatchZoo

Facilitating the design, comparison and sharing of deep text matching models.
Apache License 2.0
3.83k stars 899 forks source link

use_multiprocessing and save_model from callback #752

Closed mikelam14 closed 3 years ago

mikelam14 commented 5 years ago

Describe the Question

I am using the EvaluateAllMetrics callback, and fit_generator to evaluate at epoch end. The model_save_path works fine if i use 1 worker and use_multiprocessing=False. How do I enable multiprocessing while saving model weights? Thank you evaluate = mz.callbacks.EvaluateAllMetrics( model, x=pred_x, y=pred_y, batch_size=20, model_save_path='runs/exp')

Describe your attempts

uduse commented 5 years ago

The saving mechanism in the callback was not implemented with concurrency in mind. What exactly is the problem you're having right now? If you are referring to simultaneous writing to the same path, then currently there's no way to avoid that.

mikelam14 commented 5 years ago

So is there a way to run the training in parallel and save weights after each epoch, maybe with another callback?