give different results. In the same discussion @mcabbott said
I suspect this is more an accident than a design, but I'm not sure it's an awful one.
If you are doing ordinary AD and happen to get an array of zeros on some batch, probably you do want that to update the momenta etc.
But you won't get nothing just because of the data in that batch. Instead, you'll get it because you are e.g. doing transfer learning, or the generator & discriminator on even/odd steps, or something like that. You will get nothing not for one array, but for a whole part of the model. And it seems like you probably don't want to update the momenta for the part of the model not being trained, but instead just ignore them completely.
but i think these examples should correspond to the opt_tree only having part of the model or to using different trees for discriminator and generator.
So in this issue I argue we should treat nothing exactly as semantically equivalent to a zero gradient, and define another type e.g. NoUpdate to signal that the apply! rule should not be called at all (so no momentum updates etc...)
As mentioned in https://github.com/FluxML/Optimisers.jl/pull/137#discussion_r1159911990, when a
nothing
gradient is encountered theapply!
rule is not called at all and the state is not updated. So these two callsgive different results. In the same discussion @mcabbott said
but i think these examples should correspond to the opt_tree only having part of the model or to using different trees for discriminator and generator.
So in this issue I argue we should treat
nothing
exactly as semantically equivalent to a zero gradient, and define another type e.g.NoUpdate
to signal that theapply!
rule should not be called at all (so no momentum updates etc...)