search

lhnguyen102 / cuTAGI

CUDA implementation of Tractable Approximate Gaussian Inference
MIT License
30 stars 9 forks source link

cuda bug - Batchnorm #88

Open jamesgoulet opened 1 week ago

jamesgoulet commented 1 week ago

@lhnguyen102 I noticed that when running the MNIST example in classification.py with the FNN_BATCHNORM architecture, it runs fine on CPU but not on GPU.

Note that here I have remove the cap_factor in order to identify the source of the issue. If I use a cap_factor>>1, it works for CPU and GPU.

You can see the example on my branch: https://github.com/lhnguyen102/cuTAGI/tree/norm_layer_bug_JG

lhnguyen102 commented 6 days ago

Update: @jamesgoulet just run the same test in the C++ instead of python. It worked without any issue

################
Epoch #3/10
Training...
        Error rate for last 100 observation: 0.000
        Error rate for last 100 observation: 0.000
        Error rate for last 100 observation: 0.020
        Error rate for last 100 observation: 0.010