Mixed precision run-time error (Differential revision D21371575)

[desync-securax]

Bug Description

While trying to start training, using the AMP, the following run-time error is being induced: _Epoch 1 started! terminate called after throwing an instance of 'std::invalidargument' what(): Variable::addGrad: attempted to add child gradient of type f16 to a Variable of type f32. You might be performing an operation with two inputs of different types. Note, that there are no issues with the project compilation.

NB: This bug is reproduceable only on differential revision D21371575 by @jacobkahn

Reproduction Steps

• Any configurations or custom setup
  • Configuration file - https://pastebin.com/c08Fcmte
  • Architecture - https://pastebin.com/gWMxvRPW (WallStreet Journal)
  • ArrayFire version - 3.71 (ArrayFire-v3.7.1_Linux_x86_64)
  • Flashlight version - Differential revision D21371575
• Commands to run mpirun --allow-run-as-root -n 8 ./Train train --flagsfile=config/conv_glu/wsjtrain_neww2l.cfg --logtostderr=1 --stderrthreshold=0 --reportiters=500 --enable_distributed=1 --mixedprecision --reducer_dtype
• Trace of the error https://pastebin.com/amtYc7RM

Platform and Hardware

Hardware: 36-core Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz / 192GB RAM / 8x NVidia GTX 2080Ti 12GB VRAM OS: Linux Ubuntu 18.04 LTS Compiler: GNU C and C++ compilers 7.5.0

Additional Context

• This bug cannot be reproduced with the original version from mtmd (and related flashlight and ArrayFire versions).
• We also tested with the _am_500ms_future_context_wav2vec.arch_ architecture (additionaly to second line in the architecture, we added the following - 'PC f16'). The results are the same.

[jacobkahn]

@desync-securax — thanks for reporting. I'm still ironing out operators in which we're doing implicit casts (and making these explicit) given some changes to the AMP code. I'm hoping to commit all of these changes in the next week or two. Note that these will differ from the original changes from mtmd in that operators will be strongly typed at forward time, meaning you won't get cryptic errors like this when calling backward.

Let me know if you have any other questions. I'll leave this issue open to track this as I start committing those changes. For now, you can continue using mtmd's branches as needed.