search

nnstreamer / nntrainer

NNtrainer is Software Framework for Training Neural Network Models on Devices.
Apache License 2.0
135 stars 71 forks source link

[ Loss Scale ] Enable to define different type of Variable and Gradient for Loss Scale #2562

Closed jijoongmoon closed 2 months ago

jijoongmoon commented 2 months ago

In this PR

This PR enables separate definition of variable and gradient of Weight. It's because we have to support different type considering Mixed Precision Training or Quantization.
For example, Variable should be FP16 and Gradient should be FP16 for FP16-FP16 Mixed Precision. Or Variable should support BCQ3 and Gradient FP16 for BCQ3-FP16 QAT. Therefore, we need to set update the Variable weight as it is requested.. and for Gradient, we have to set same type of Activation.

This PR enables to set different type between Variable and Gradient.

Commits to be reviewed in this PR

[Property] Add loss scale property
It add loss scale property as model common property. Signed-off-by: Jiho Chu
[ Weight ] Add Loss Scale factor in Weight
This PR enables the loss scale factor in Weight. . Change the WeightSpec to incluide the loss factor . Add LossScaleForMixed Property as an layer common property, so that it can set the scale factor in initContext. . Add Loss Scale in initContext . Set the LossScaleForMixed Property when there is LossScale Model Property Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon
[ Weight ] split variable dim and grad dim to set separately
This PR split the Variable and Gradient Dim in Var_Grad and Weight. By this way we can set different Variable Type and Gradient in Wegiht. . add dim_g for gradient in WeightSpec. . manager need to update to support WeightSpec. . Create Tensors according to dim_v and dim_g . Create Weight chaged in Weight.h Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon
[ NEURALNET ] change the loss scale property to Rigid Property
Loss Scale is more like Rigid Property of model, rather than flexible property. Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon
taos-ci commented 2 months ago

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2562. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.