[EPIC] Training Element (a.k.a. tensor_training)

[myungjoo]

NOTE

• We need a topic owner.

  • You need to be already a veteran nnstreamer developer
  • You will need assistance from nntrainer, too. (need to study how it works)
  • You will need to understand usage scenarios of on-device / among-device AI training.

• Timeline

  • 22'2Q: background research (don't take this with full load)
  • 22'3/4Q: PoC / First Release
  • May be updated a lot according to production plans.

Objective

Allow to train & update neural network models within nnstreamer pipelines, starting w/ nntrainer and extensible for other frameworks including TF, Keras, and PyTorch.

Name it!

• tensor_train?
• tensor_learn?
• ...

Usage Case Scenario

• Normal training with given stream input
  • Some tensor stream (num_tensors=2, supervised) ! tensor_train ... data=0 label=1 mode=sink
• Reinforcement training
  • separated training & inference streams?
    • Then, how do we synchronize models between tensor_filter and tensor_train?
  • integrated stream?
• Remote training (nnstreamer-edge & ml-service-api Phase4+Phase2)
• How do we update other instances of tensor-filter & single-API with the trained (upated) model?
  • After the training?
  • During the trainig?
• Stop/Continue/Terminate w/ pipelined training?

Tasks

Let's determine after "Usage Case Scenario". However, the following trivial subtasks can be identified:

• [x] Core Implementation
• [ ] Subplugin Implementation
  • [x] NNTrainer (must & 1st)
  • [ ] TF (hobby prj)
  • [ ] Keras (hobby prj)
  • [ ] PyTorch (hobby prj)
• [x] Example Apps
• [ ] Documentation
• [ ] Language & APIs
  • [ ] C API
  • [ ] .NET API
  • [ ] Web API
  • [ ] Java/Android support

PTAL: @nnstreamer/nnstreamer / Add more usage case scenarios (for better requirement and feature listing)

This obsoleted #1715

[taos-ci]

:octocat: cibot: Thank you for posting issue #3745. The person in charge will reply soon.

[myungjoo]

Draft 220602_MJ_A (WIP)

Filter Name: tensor_trainer Subplugin Required: true

• mode = separated streams

  • SINKPAD (input) 0: X (input tensors)
  • SINKPAD (input) 1: y (supposed output)

• mode = aggregated stream

  • SINKPAD (input) 0: X, y (designate which are 'supposed output' with properties)

• properties

  • mode (as shown above)
  • input_tensors=.... (may reorder tensors of input tensors) required if mode=aggregated stream, otherwise, optional
  • answers=.... (may reorder tensors of supposed output) required if mode=aggregated stream. otherwise, optional
  • batchrank=-1...MAXRANK-1 (rank of dimension (input/answer) that designates index of a batch. -1 if there is no batch.
  • batchsize=1...MAXINT

[songgot]

I want to do this.

[myungjoo]

Please write proposals of drafts as usage cases of pipeline topology (component view). We may need to include a repository/database element for training data. (e.g., GST-Pravega w/ other/tensors supprot)

[myungjoo]

Milestone

• Design iteration: 2023.1.
• Working-PoC: 2023-1Q
• Release with basic features: 7.5 M1
  • nntrainer compatibility
  • C-API
  • example pipelines & apps
• Release with full features: 7.5 M2
  • general framework compatibility (at least w/ subplugins of pytorch, tensorflow)
  • more features for faster/more-efficient training (DB / data repository in pipeline)
  • C#-API
  • Web-API (depending on SRPOL's schedule)
  • more realistic (practical) sample pipelines & apps.
  • example apps registered in Tizen Studio.

[myungjoo]

@songgot, please identify Public APIs required for tensor_trainer in pipelines and datarepo src/sink. You don't need to add APIs for training in general, that's for ML Training API. And register C, .NET APIs for Tizen before 9.0 M1. And Web APIs for Tizen 9.0 M2.