[gfx10] ConvOclBwdWrW1x1 fails

[zjing14]

ROCm 4.0, Nav21

Fp32

MIOpenDriver convfp16 -V 1 -F 0 -n 256 -c 128 -H 28 -W 28 -k 512 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -t 1 
MIOpen Backward Weights Conv. Algorithm: 1, Solution: 25/ConvOclBwdWrW1x1 
Backward Convolution Weights Failed: 0.163441 > 0.0164 

MIOpenDriver convfp16 -V 1 -F 0 -n 256 -c 256 -H 56 -W 56 -k 128 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -t 1 
MIOpen Backward Weights Conv. Algorithm: 1, Solution: 25/ConvOclBwdWrW1x1 
Backward Convolution Weights Failed: 0.103369 > 0.0164 

Fp16

MIOpenDriver convfp16 -V 1 -F 0 -n 256 -c 256 -H 56 -W 56 -k 128 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -t 1
MIOpen Backward Weights Conv. Algorithm: 1, Solution: 25/ConvOclBwdWrW1x1
Backward Convolution Weights Failed: 0.103369 > 0.0164

[atamazov]

Please fix typo

[atamazov]

@junliume I am afraid that this should be reopened (recommended labels: https://github.com/ROCmSoftwarePlatform/MIOpen/labels/urgency_normal https://github.com/ROCmSoftwarePlatform/MIOpen/labels/quality https://github.com/ROCmSoftwarePlatform/MIOpen/labels/performance https://github.com/ROCmSoftwarePlatform/MIOpen/labels/workaround)

Right now we have WORKAROUND_SWDEV_266868. I think that at first we need to make the decision to either fix the issue or convert W/A into a permanent part of the solver's design.

• If we would like to make it permanent, then we need a PR to implement that (quality fix).
• Otherwise we have to continue investigations -- find out the root reason and fix, if possible (performance fix)

I would like to mention two circumstances that may affect the decision.

• We still don't know what exactly causes the accuracy error. Therefore, there is a risk that problems with this solver will also occur on other GPUs.
• The performance drop is substantial (2X on average for the known problematic configs). Results on Navi21, ROCm 5.3.0:

Config	Without W/A	With W/A	Relative performance
convfp16 -F 4 -n 256 -c 128 -H 28 -W 28 -k 512 -y 1 -x 1 -p 0 -q 0	3.1	5.2	0.60
convfp16 -F 4 -n 256 -c 256 -H 56 -W 56 -k 128 -y 1 -x 1 -p 0 -q 0	5.9	13.7	0.43
conv -F 4 -n 256 -c 128 -H 28 -W 28 -k 512 -y 1 -x 1 -p 0 -q 0	6.3	15.2	0.41
conv -F 4 -n 256 -c 256 -H 56 -W 56 -k 128 -y 1 -x 1 -p 0 -q 0	12.4	23.4	0.53

[atamazov]

@junliume I recommend removing https://github.com/ROCmSoftwarePlatform/MIOpen/labels/bug and https://github.com/ROCmSoftwarePlatform/MIOpen/labels/correctness -- that would better match the current status of the issue.

[ppanchad-amd]

@atamazov Is this fixed in ROCm 6.0.2? Thanks

[atamazov]

@ppanchad-amd Not fixed. It is necessary to further investigate the problem, so some engineer must be assigned.

@junliume I recommend adding [gfx11] to the title as WORKAROUND_SWDEV_266868 affects that target as well as gfx10.