Fixing wandb defaults and default run naming scheme
Keeping all PSD plots from across training in W&B run
Fixing best_model_path in checkpoint callback to use min for monitoring
Loading in best weights before testing
Saving X_scaler and y_scaler parameters as torch modules at the end of training
You can see results from a successful run of this model here. This should bring the training from this repo into full parity with the origina deepclean_prod training.
IMPORTANT NOTE: You'll want to load the best validation weights in at test time, right now it looks like you're just taking last.ckpt but this isn't what you want. You have a few options here:
Right now train.callbacks.ModelCheckpoint.on_train_end loads in the best weights and uses this to build a compiled torch graph. Instead, you could just copy the best weights to something like best.ckpt and just look for that
You could use W&B's API at inference time to infer the best checkpoint using something like the suggestion outlined here
SECOND IMPORTANT NOTE: These changes came in parallel to #8 , which in light of these fixes probably should be reverted before merging since it incorrectly diagnosed the source of issues we were seeing with test performance.
best_model_pathin checkpoint callback to useminfor monitoringX_scalerandy_scalerparameters as torch modules at the end of trainingYou can see results from a successful run of this model here. This should bring the training from this repo into full parity with the origina deepclean_prod training.
IMPORTANT NOTE: You'll want to load the best validation weights in at test time, right now it looks like you're just taking
last.ckptbut this isn't what you want. You have a few options here:train.callbacks.ModelCheckpoint.on_train_endloads in the best weights and uses this to build a compiled torch graph. Instead, you could just copy the best weights to something likebest.ckptand just look for thatSECOND IMPORTANT NOTE: These changes came in parallel to #8 , which in light of these fixes probably should be reverted before merging since it incorrectly diagnosed the source of issues we were seeing with test performance.