Open johanna-rock-tt opened 4 months ago
Current perf breakdown: https://docs.google.com/spreadsheets/d/1YCIqrqJ6cPd0AVZs1DiXuEOOrPL9boYmEIFSDufKj6M/edit?usp=sharing Branch: main@61b3856b7d42daf1888604b53a6090733e52f5be System: t-3012 @ 1GHz
Most costly ops for S=2k
LN accumulated (slices): 3.8% per layernorm
Most costly ops for S=128
fyi @pavlejosipovic @s-jovic @pavlepopovic @uaydonat
After couple of weeks and several changes there is new perf burndown:
milestone | single layer latency (ms) | full model latency (ms) | t/s | notes |
---|---|---|---|---|
current performance (bfloat16) | 19.71 | 1182.8 | 1731 | May 14th, main, bidirectional AllGather |
target @50% | 5.47 | 328.2 | 6240 | |
nvidia benchmark | 10.9 | 654.6 | 3128 | |
projected after burndown (bfloat) | 11.3 | 684 | 2991 |
Breakdown per type
Type | % | Time(ns) |
---|---|---|
AllGather | 48.5 | 9.57 |
Matmuls | 35.9 | 7.08 |
Layernorm | 6.7 | 1.31 |
milestone | single layer latency (ms) | full model latency (ms) | t/s | notes |
---|---|---|---|---|
current performance (bfloat16) | 2.24 | 134.5 | 951.6 | May 16th, main, biidirectional AllGather |
target @50% | 0.83 | 50 | 2560 | |
nvidia benchmark | 1.4 | 84.04 | 1523 | |
projected after burndown (bfloat16) | 1.68 | 101.4 | 1262 |
Breakdown per type
Type | % | Time(ns) |
---|---|---|
AllGather | 30.9 | 0.69 |
Matmuls | 56.5 | 1.27 |
Layernorm | 5.9 | 0.13 |
Next steps for seq_len=2048:
I believe the tasks should be executed in respective order to optimize model in areas where the largest impact can be made according to breakdown.
We already tested casting activations for AllGather to bfp4_b
and that resulted in bad pcc (already for one layer), but using bfp8_b
gave good pcc.
e2e perf issue: #8866
Tracks the open issues for Falcon40b prefill to hit target perf.
Last updated: May 27th
Prefill
bfp8
Seqlen=2048
1 layer currently takes 12.54 ms One-time ops: + 2.2 ms
Open Issues (bfp8 version):
Target estimates see: https://docs.google.com/spreadsheets/d/1LawF5YIbAQC1c7vMJG7z-_qh1YXojpnWNvgLMBv5jC8 Assuming HiFi2, S=2048 has a target of 6240 tok/sec
Seqlen=128
Assuming same optimizations as above. 2k is the harder regime, but optimizations should also apply to seqlen=128.
1 layer currently takes 1.80 ms One-time ops: + 0.87 ms
WIP Projections:
Assuming HiFi2 here, then S=128 has a target of 1423 tok/sec