Closed jjsjann123 closed 1 week ago
cc'ing @wujingyue :sob:
Apparently, the bug predated https://github.com/NVIDIA/Fuser/commit/fe34321ad0a36a6fe96a625b3dc81e1ed716ef1b. I'm running a git bisect
...
Bisect hit 1c80008afd1719dc3e36ccafc0294e9cb73a8b70, which broke build. I'll try to git bisect skip
commits between that and 0cc22b910b5a6302444850019ae1e4f92974bc83...
https://github.com/NVIDIA/Fuser/commit/2c3c08fa994286018aa3a53aefc201e49fb26593 is the first commit that broke the reproducer. cc @liqiangxl
Seems due to bug in sortAndRFactor
, the correct order should be
reduction_rf_tv: T76_l[ ..., rS459{( ceilDiv(( ceilDiv(4, 2) ), 1) )}rf, iUS456{1}, iV452{8}, iUS460{1}rf, rUR458{2}rf ]
instead of
reduction_rf_tv: T76_l[ ..., iUS456{1}, iV452{8}, rS459{( ceilDiv(( ceilDiv(4, 2) ), 1) )}rf, iUS460{1}rf, rUR458{2}rf ]
During lowering, ceilDiv(4, 2)
is replaced with ceilDiv(ceilDiv(T4.logical_size[1LL], 8), 2);
ceilDiv(4, 2)
also should be symbolic instead of const. Not sure why ceilDiv(i14, 8)
is chagned to 4
in T53_g = __bfloat2float(T70_l)
Inputs:
T4_g[ bS12{1}, iS13{i14}, iS412{8192}, iS413{128} ], __bfloat
Outputs:
T57_g[ bS256{1}, iS394{8}, bS258{1}, iS395{8192}, iS396{128} ], __bfloat
%kernel_math {
T70_l[ bS325{1}, iS330{8}rf, iS331{( ceilDiv(i14, 8) )}rf, iS414{8192}, iS415{128} ] = view( T4_g[ bS12{1}, iS13{i14}, iS412{8192}, iS413{128} ] )
T53_g[ bS372{1}, iS373{8}, iS374{4}, iS375{8192}, iS376{128} ]
= __bfloat2float(T70_l[ bS325{1}, iS330{8}rf, iS331{( ceilDiv(i14, 8) )}rf, iS414{8192}, iS415{128} ]);
T71_g[ iS377{8}, iS378{4}, iS379{8192}, iS380{128} ]
= squeeze( T53_g[ bS372{1}, iS373{8}, iS374{4}, iS375{8192}, iS376{128} ] )
T72_l[ iS381{8}, rS382{4}, iS383{8192}, iS384{128} ]
= reduction( T71_g[ iS377{8}, iS378{4}, iS379{8192}, iS380{128} ], op = add, initial value = float(0), allreduce = false )
T55_g[ iS388{8}, iS389{8192}, iS390{128} ]
= __float2bfloat(T72_l[ iS381{8}, rS382{4}, iS383{8192}, iS384{128} ]);
T56_g[ bS251{1}, iS391{8}, bS253{1}, iS392{8192}, iS393{128} ]
= broadcast( T55_g[ iS388{8}, iS389{8192}, iS390{128} ] )
T57_g[ bS256{1}, iS394{8}, bS258{1}, iS395{8192}, iS396{128} ]
= Set( T56_g[ bS251{1}, iS391{8}, bS253{1}, iS392{8192}, iS393{128} ], cache_op=Streaming )
} // %kernel_math
In ExactLogicalDomainMap
, we have
{ iS331{( ceilDiv(i14, 8) )}rf; iS374{4}; iS378{4}; rS382{4} }
We should be able to derive i14 = 32
, so we don't need to replace i14
with T4.logical_size[1LL]
, this can also fix the bug. what do you think @jacobhinkle ?
Not sure why
ceilDiv(i14, 8)
is chagned to4
inT53_g = __bfloat2float(T70_l)
I think it's the other way around: the 4
exists in the dynamic fusion before dynamic shape concretization:
T50_l[ ?S219{1}rf, ?S220{8}rf, ?S221{4}rf, ?S222{8192}rf, ?S223{128}rf ] = view( T4_g[ bS12{1}, iS13{i14}, iS14{i15}, iS15{i16} ] )
T53_l[ ?S238{1}, ?S239{8}, ?S240{4}, ?S241{8192}, ?S242{128} ]
= __bfloat2float(T50_l[ ?S219{1}rf, ?S220{8}rf, ?S221{4}rf, ?S222{8192}rf, ?S223{128}rf ]);
When we concretize T50_l
's root->logical transforms, the ID ?S221{4}rf
is replaced with the output of a Split
, which has the ceilDiv
extent.
Can someone summarize? Do we need both #2713 and #2714 or just #2714 which is being worked on? I guess the latter but wasn't sure.
Can someone summarize? Do we need both #2713 and #2714 or just #2714 which is being worked on? I guess the latter but wasn't sure.
You got it right. It's only #2714 that's needed. cc'ing @liqiangxl should we close #2713 ?
Seems to be a functional regression. A new failure show up from disabling bookend in the qkv_split_rope backward.
Error message:
vvv