Closed finncatling closed 3 years ago
I have emailed the BDAU to see if our script is causing any issues. I'll also write some temporary code to save the NovelModel instance after every split of model training
Now running in the BDAU using nohup
, with the script output logging to outputs/7_and_8_testing.log
Script crashed during training on train fold 103/120. It's not clear what caused the crash but doesn't look like a python error (there was no error message in the nohup
log). Luckily, we had modified 08_train_eval_novel_model.py
to save NovelModel
after each split iteration by externalising the training loop usually implemented in NovelModel.fit()
:
for split_i in pb(
range(novel_model.cat_imputer.tts.n_splits),
prefix="Split iteration"
):
novel_model._single_train_test_split(split_i)
save_object(
novel_model,
os.path.join(NOVEL_MODEL_OUTPUT_DIR, "08_novel_model.pkl"))
so it was easy to modify the script to resume training as follows:
novel_model: NovelModel = load_object(
os.path.join(NOVEL_MODEL_OUTPUT_DIR, "08_novel_model.pkl"))
for split_i in pb(
range(len(novel_model.models) - 2, novel_model.cat_imputer.tts.n_splits),
prefix="Split iteration"
):
novel_model._single_train_test_split(split_i)
save_object(
novel_model,
os.path.join(NOVEL_MODEL_OUTPUT_DIR, "08_novel_model.pkl"))
...
This completed the training without issue.
This has happened twice now.