Offline mode for wandb

[krisrs1128]

• gives dry run option, for running offline
• moves parameter counting to after the definition of self.d and self.g
• uploads runs in the wandb directory to the cloud

Some warnings about the last point,

• While the offline saving works, the final upload can't be done on compute canada... because the run is offline. You could always sync after the sbatch jobs finish (again using wandb sync) but then I'm not sure why we have these lines 526 - 531 in the first place.
• It will sync all the runs in the output/wandb folder. I think that's fine, but if you would prefer only uploading specific run IDs, that can be built in.

[vict0rsch]

what you're saying about the end of the script is true, we could just do it manually

[krisrs1128]

I removed the offline upload, because I think the sbatch jobs might get unnecessarily prolonged. Comet throws an error if it can't upload, but wandb keeps making attempts.

If it's too annoying to do by hand, we can have a separate upload script.