nnstreamer / nntrainer

NNtrainer is Software Framework for Training Neural Network Models on Devices.

Apache License 2.0

135 stars 71 forks source link

Normalization Layer: FP32/FP16 should NOT be determined at compile-time #2408

Open myungjoo opened 6 months ago

myungjoo commented 6 months ago

Having calculations based on fp16 or fp32 should be determined at run-time based on the model and user's intention, not by the compiler-options.

https://github.com/nnstreamer/nntrainer/blob/0083caa3f5ebccba2512590e740d056a318e43bc/nntrainer/layers/layer_normalization_layer.cpp#L207-L225

taos-ci commented 6 months ago

:octocat: cibot: Thank you for posting issue #2408. The person in charge will reply soon.

jijoongmoon commented 6 months ago

People recommand to use FP32 computation for normalization layers like batch normalization, layer normalizaiton or rms normalization, etc. We will check once again.

Deeksha-20-99 commented 3 months ago

Hello could you let us know the progress on this issue.. Since we (from ASU) are also working and facing the same issue.

myungjoo commented 3 months ago

Hello could you let us know the progress on this issue.. Since we (from ASU) are also working and facing the same issue.

It's not refactored, yet. You will need to choose it with build-option for this version. I guess we will update these after the next product release.

myungjoo commented 3 months ago

Hello could you let us know the progress on this issue.. Since we (from ASU) are also working and facing the same issue.

2549 may be interesting to you.