xiezipeng-ML
commented
2 years ago batch size = 4, acc step = 8, amp, open Checkpointing
| 1n1g | use_fuse_mask_softmax = False | use_fuse_mask_softmax = True |
|---|---|---|
| Throughput | total_throughput: 152.35 samples/s | total_throughput: 158.00 samples/s |
| GPU Memory | 3145MiB | 3335MiB |
| 1n4g | use_fuse_mask_softmax = False | use_fuse_mask_softmax = True |
|---|---|---|
| Throughput | total_throughput: 109.33 samples/s | total_throughput: 112.39 samples/s |
| GPU Memory | 2445MiB | 2545MiB |
@chengtbf @strint @ouyangyu
测试use_fuse_mask_softmax 的性能增益
oneflow分支：python3 -m pip install --pre oneflow -f https://staging.oneflow.info/branch/release/mt5_opt/cu112
对应的oneflow commit：2d080aa
libai分支：use_fuse_mask_softmax
在
projects/T5/configs/t5_model_config.py中测量model.cfg.scale_mask_softmax_fusion = False和model.cfg.scale_mask_softmax_fusion = True上的吞吐区别@ouyangyu @chengtbf @strint