Transfer current models/training from notebook to script

raspstephan / nwp-downscale

MIT License

38 stars 9 forks source link

Transfer current models/training from notebook to script #36

Closed HirtM closed 3 years ago

HirtM commented 3 years ago

[x] module with different Generators/discriminators
[x] corresponding training routines
[x] script for training that saves trained model etc.
[x] test training script
[ ] save output properly (currently not properly working)
[ ] make option to save generator at intermediate epochs
[ ] loss functions, mse, ... need to be saved
[x] make subdirectory for each experiment in saved models?
[ ] plot examples at intermediate epochs to watch training progression? (save them in folder?)
[ ] document training progression
[x] logging file

HirtM commented 3 years ago

It seems to me that the argparser can not handle boolean input, instead one needs to use options like --no-sigmoid. Needs to be modified for all boolean values.

HirtM commented 3 years ago

With the training script (I modified the existing train.py script), the generator and discr. seem to be successfully saved with : torch.save(gen, save_path) (no error messages)

However, I am not able to read in the models again with gen = torch.load(gen_file): EOFError: Ran out of input

I also tried changing maplocations to cuda/cpu, but that didn't change anything.

@raspstephan Do you have an idea what is going wrong?

HirtM commented 3 years ago

With the training script (I modified the existing train.py script), the generator and discr. seem to be successfully saved with : torch.save(gen, save_path) (no error messages)

However, I am not able to read in the models again with gen = torch.load(gen_file): EOFError: Ran out of input

I also tried changing maplocations to cuda/cpu, but that didn't change anything.

@raspstephan Do you have an idea what is going wrong?

Can be fixed by saving state.dicts instead.

HirtM commented 3 years ago

@raspstephan I'll wait with the rest of the Todo's here for now, since you said that the notebooks may be more handy anyway, at least for now.

raspstephan commented 3 years ago

@raspstephan I'll wait with the rest of the Todo's here for now, since you said that the notebooks may be more handy anyway, at least for now.

Yes, I think that's probably best for now. I think we should first do some manual tuning in notebooks to get something working.

raspstephan commented 3 years ago

Closing this for now since we will work in notebooks for now. Thanks again @HirtM for putting our enhancements into the modules :)