Fix Layer Norm - Githubissues

lhnguyen102 / cuTAGI

CUDA implementation of Tractable Approximate Gaussian Inference

MIT License

24 stars 9 forks source link

Fix Layer Norm #64

Closed lhnguyen102 closed 2 months ago

lhnguyen102 commented 2 months ago

Description

This PR added a fix for layer norm issue mentioned in #60.

Changes Made

Removed the running mean and variance for layer norm in both CUDA and CPU layer_norm.cpp and layer_norm_cuda.cu
Adapted Python API with the changes in the back-end

Note for Reviewers

Currently we haven't released this fix yet. Here are the steps to test latest changes

Remove pytagi package in your conda environment if exisits
```
pip uninstall pytagi
```
Compile C++/CUDA code using cmake

Added the following temporary import to the beginning examples/classification.py

# Temporary import. It will be removed in the final version
import os
import sys

# Add the 'build' directory to sys.path in one line
sys.path.append(
    os.path.normpath(os.path.join(os.path.dirname(__file__), "..", "build"))
)

Run classification examples
```
python -m examples.classification
```
NOTE: I have to increase sigma_V to 2.0 form 1.0 for a larger batch size, otherwise it is not stable during the training

close #60