search

xuhuisheng / rocm-build

build scripts for ROCm
Apache License 2.0
181 stars 35 forks source link

invalid output with gfx803 and BUILD_WITH_TENSILE_HOST ON #4

Open xuhuisheng opened 3 years ago

xuhuisheng commented 3 years ago

Finally, I can reproduce gfx803 with BUILD_WITH_TENSILE_HOST errors using a small sample.
OS Ubuntu-20.04.1
linux Linux 5.4.0-64-generic
ROCm 4.0.1
GPU RX 580 8G
pytorch 1.7.1

test codes : https://github.com/xuhuisheng/rocm-build/blob/feature/check/check/test-pytorch-rocblas.py In this code, there is one full connection layer Y = w0 * x0 + w1 * x1 + b

var value
input num 2
output num 1
weight0 1
weight1 1
bias 10
features 800
batch 32

For simple case, I init weight with [1,1], bias with 10, so it should not change anything, loss always should be 0. But on the 12th steps, loss changed to 50. It is reproducable on my computer.

I print the X, Y, output, loss, and find out that the first 16 of output is 12 which is correct. the last 12 of output is 2 which is wrong. And seems the last 16 of output miss the bias, I guess.

     X tensor([[1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.]], device='cuda:0')
     Y tensor([1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01], device='cuda:0')
weight tensor([[1., 1.]], device='cuda:0')
  bias tensor([10.], device='cuda:0')
output tensor([[1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00]], device='cuda:0', grad_fn=<AddmmBackward>)
     l tensor(50., device='cuda:0', grad_fn=<MseLossBackward>)

If I used BUILD_WITH_TENSILE_HOST=OFF and re-compile rocBLAS, the error will gone.

And It is weired if I comment out l.backward(), the error will change to happen on 21th step, and output change to all 3. Why it is unstable? Maybe gfx803 need PCI3 Atomics operation? or gfx803 just miss some waitcnt() instructions?

     X tensor([[1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.]], device='cuda:0')
     Y tensor([12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000,
        12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000,
        12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000,
        12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000],
       device='cuda:0')
weight tensor([[1., 1.]], device='cuda:0')
  bias tensor([10.], device='cuda:0')
output tensor([[3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.]], device='cuda:0', grad_fn=<AddmmBackward>)
     l tensor(81., device='cuda:0', grad_fn=<MseLossBackward>)

So I believe it caused by the new Tensile client which used assembly codes to optimized. I am not familiar with GCN assemly, So I create an issue to tensile wish someone could help. https://github.com/ROCmSoftwarePlatform/Tensile/issues/1287

xuhuisheng commented 2 years ago

Thank @https://github.com/cgmb to show how to use rocblas-test to verify the issue of rocblas on gfx803. https://github.com/ROCmSoftwarePlatform/rocBLAS/issues/1218. Keep investigating.