search

NVIDIA / nv-wavenet

Reference implementation of real-time autoregressive wavenet inference
BSD 3-Clause "New" or "Revised" License
735 stars 126 forks source link

Problem with testing script #4

Closed julianzaidi closed 6 years ago

julianzaidi commented 6 years ago

Hi,

I have tried to run the test script from the Makefile with the command make nv_wavenet_test ./nv_wavenet_test. Everything works correctly and the test runs. However, when I reach the large WaveNet model (R=64 -- S=256), an assertion occurs when the code compares the matrices obtained with the reference and the CUDA implementation.

At a given layer in the model, the difference between the two matrices is greater than the epsilon value of the function matrix_compare(name, A, B, epsilon). I can increase this value in the code and the assertion will disappear, but I would like to understand why this error.

Here is my GPU configuration : NVIDIA 1080Ti - CUDA 9.0 - Driver version: 384.111

Thanks a lot

julianzaidi commented 6 years ago

Edit to my previous problem

I decreased batch_size parameter and there is no more assertion error when I run the test

BrianPharris commented 6 years ago

We currently don't use much sophistication in selecting random weights in the test program. Looks like we just got unlucky with this particular invocation. Closing, but please reopen if you encounter more consistent failures.