LibCEED failures on CI node

CHIP-SPV / chipStar

chipStar is a tool for compiling and running HIP/CUDA on SPIR-V via OpenCL or Level Zero APIs.

Other

226 stars 33 forks source link

LibCEED failures on CI node #857

Open pvelesko opened 5 months ago

pvelesko commented 5 months ago

Some tests fail correctness when using /gpu/hip/ref, these tests do not fail when using /gpu/hip/shared or /gpu/hip/gen. On cupcake , none of these tests fail at all.

The following tests fail:

Test Summary Report
-------------------
t352-basis          (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t506-operator       (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1

√ Updated the runtime to match cupcake √ Fails for both OpenCL and Level Zero √ Passes on PoCL

pvelesko commented 2 months ago

These failures seem to be related to the runtime. Furthermore, these tests do not fail on ALCF systems which use engineering SDK drops.

If the problem size is reduced, I don't see these tests failing anymore.
Using shared memory also makes the error go away
PoCL also passes

It's very difficult to extract a reproducer from these so I think we can revisit during 1.3 as these don't seem to be an issue in our implementation.