captainvera
commented
4 years ago Hello @HanchengDeng,
The behaviour you're describing is unexpected. There's no extra operation to make the trained model work on a new machine.
What could be happening is an unintended consequence of the saving behaviour we implemented in OpenKiwi. You see, when you're training a model and you run the validation scoring the model is instantly saved if it is one of the best runs. If that is the case, in order to avoid using too much disk space, we save the best_model.torch as a path to you're actual best model (which will have a different name). So if the best model is the exact last validation, this best_model will only be a link to the actual file where the model is saved. You can check the behaviour on trainers/callbacks.py save_latest.
Of course, this works when you're in the same machine where you have all the other checkpoints saved but won't work when you move to a new server.
Can you check if this is the case?
HanchengDeng
Hello! @captainvera
I trained a PredEst model on a server, and I could use the "kiwi predict --config predict.yaml" to make a prediction perfectly.
Recently, I wanted to use the trained PredEst model to make a prediction on another new server, and I just moved the trained best_model.torch from the old server to the new one to make it work. However, it does not work and showed
[kiwi.lib.predict run:100] Predict with the PredEst (Predictor-Estimator) model Killed. And it seems that the process[kiwi.data.utils load_vocabularies_to_fields:126] Loaded vocabularies from models/best_model.torchis missing.I am wondering if there is something wrong or missing with my operation to make the trained model work on the new server. Looking forward to your response. Thank you very much!