[onert-micro] Introduce TrainingConfigureTool

Samsung / ONE

On-device Neural Engine

Other

438 stars 157 forks source link

[onert-micro] Introduce TrainingConfigureTool #13684

Open BalyshevArtem opened 3 months ago

BalyshevArtem commented 3 months ago

Let's introduce training configure tool. This tool should find a configuration for network training with some optimal memory costs. The Training Tool will be able to perform actions to improve the training process:

based on a given memory budget, select only a part of the layers for training, or even a part of the weights of a certain layer - this process called sparse backpropagation.
applications of mixed precision quantization, search for parts of the network for which the materialization technique will be performed (when intermediate results are not saved for some part of the network, but they are recalculated in the backpropagation process)
optimizations on the graph itself
and so on

BalyshevArtem commented 3 months ago

draft - https://github.com/Samsung/ONE/pull/13479

BalyshevArtem commented 3 months ago

Result of the tool from https://github.com/Samsung/ONE/pull/13479 for our target model: Default training - training all network, all operations, and all tensors:

CROSS_ENTROPY_METRIC_RESULT = 0.187580
PEAK_MEMORY_ESTIMATION_RESULT = 148 624 bytes

Training with training config file, produced by training configure tool:

CROSS_ENTROPY_METRIC_RESULT = 0.187278
PEAK_MEMORY_ESTIMATION_RESULT = 62 160 bytes