Open hubertlu-tw opened 1 year ago
Reopening
What is this blocked by ? cc: @jeffdaily @hubertlu-tw @sunway513 @dllehr-amd @aspanday
Hi all,
I've updated all files with appropriate changes and ALL fast_layer_norm tests pass for fwd and bwd now. Please let me know if there are any concerns. Thanks!
Performance uplift can be found on the "fwd + bwd all" sheet [here]
Let me know if you need access to the above link.
To run the --fast_layer_norm unit tests,
As a reference, the following results were obtained from CUDA systems (with nvcr.io/nvidia/pytorch:22.08-py3 on a A100 node):
On ROCm, it failed the following two checks with NaN outputs:
in test_fast_layer_norm.py when wtype or ctype is bf16.
However, when we skip the tests with bf16 wtype or ctype, it failed the tests starting from hidden_size=8192 due to relerr > tol. Please find the results of the unit tests here: apex_fastlayernorm_unittest.txt