Open denizetkar opened 2 years ago
I'll need some time to look into this, but I do believe that the behaviour I currently have coded is intentional.
I want the running statistics to be updated within a given episode, but then scrapped at the end if this isn't a training iteration.
@denizetkar is right, that's not your intended behaviour and the backup is overwritten by the validation pass. You simply need to use deepcopy
instead.
When
MetaBatchNormLayer
is called forward withbackup_running_statistics=True
, the running statistics are meant to be copied into the backup variables by copying:https://github.com/AntreasAntoniou/HowToTrainYourMAMLPytorch/blob/86964701651c06e490271365fedb8789a72ef3f0/meta_neural_network_architectures.py#L241
However, this is not what happens and the underlying data ends up being tied with each other such that when the running statistics get updated, so does the backup. Here is a short code snippet that minimally reproduces the same behavior:
Is my understanding of backing up the running statistics wrong or is this a bug that needs fixing?