search

spcl / dace

DaCe - Data Centric Parallel Programming
http://dace.is/fast
BSD 3-Clause "New" or "Revised" License
491 stars 124 forks source link

DaCe benchmark validation error #1128

Open clompa opened 1 year ago

clompa commented 1 year ago

I was trying to run the benchmarks to compare the numpy-dace speedup. I have good results, but there are some tests that fail for DaCe: mandelbrot1, mandelbrot2, durbin, nbody and resnet.

If I try to just run the individual benchmarks they all work as expected, eg: python run_benchmark.py -b mandelbrot1 -f dace_cpu
However I get errors if I try: python run_framework.py -f dace_cpu -p paper
This is an example of the results:

***** Testing NumPy with durbin on the S dataset *****
NumPy - default - first/validation: 9ms
NumPy - default - median: 9ms
***** Testing DaCe CPU with durbin on the S dataset *****
NumPy - default - validation: 9ms
DaCe CPU - fusion - first/validation: 71ms
DaCe CPU - fusion - validation: SUCCESS
DaCe CPU - fusion - median: 11ms
DaCe CPU - parallel - first/validation: 12ms
DaCe CPU - parallel - validation: SUCCESS
DaCe CPU - parallel - median: 11ms
DaCe CPU - auto_opt - first/validation: 9ms
Relative error: 3.424183545358736e+21
DaCe CPU did not validate!
DaCe CPU - auto_opt - median: 9ms

I have the same problem with older versions of dace, namely v0.8.1, v0.9.0, v0.10.0, v0.11.1, v0.12, v0.13 (except that mandelbrot gives another error) and with both mkl and openblas.

alexnick83 commented 1 year ago

NPBench tests 3 different optimization heuristics for DaCe, as shown above (fusion, parallel, and auto-opt). Not all of them need to work correctly for all benchmarks, although we try to fix issues (there have been quite a few updates in the latest master branch). Regarding validation, there are cases where it is too strict, especially for some iterative algorithms. We have made amendments to NPBench's validation to address this.