When using BatchNorm ops with mixed_float16, there are some overflow issues within the DirectML kernels that cause training to not converge or to generate NaN. Until this is resolve, we add a workaround to always run BatchNorm in float32 precision, and conver the result back to float16 when it's done.
When using BatchNorm ops with
mixed_float16, there are some overflow issues within the DirectML kernels that cause training to not converge or to generate NaN. Until this is resolve, we add a workaround to always run BatchNorm in float32 precision, and conver the result back to float16 when it's done.