I'm trying to restart a job by loading a network snapshot pickle (called network-snapshot-001283.pkl). The pickle is written during training of the previous run and loaded at the beginning of the restart by the functions at the beginning of training/misc.py.
I get the following error when trying to load the snapshot:
_pickle.UnpicklingError: invalid load key, '<'.
This error appears for all my snapshots produced training with different data on different runs, as well as when trying to use generate_figures.py.
Either the pickle written during training is corrupted, or I'm doing something wrong, although I haven't change any line corresponding to this part.
Any idea?
Nevermind, I've just found out that indeed the files were corrupted due to a bad file transfer between two different machines. Nothing wrong with the code.
I'm trying to restart a job by loading a network snapshot pickle (called network-snapshot-001283.pkl). The pickle is written during training of the previous run and loaded at the beginning of the restart by the functions at the beginning of training/misc.py.
I get the following error when trying to load the snapshot:
_pickle.UnpicklingError: invalid load key, '<'.
This error appears for all my snapshots produced training with different data on different runs, as well as when trying to use generate_figures.py. Either the pickle written during training is corrupted, or I'm doing something wrong, although I haven't change any line corresponding to this part. Any idea?
Thanks