Add Softplus activation function @open sesame 04/15 10:41

[heka1024]

• Now, user can use Softplus activation function like torch or tensor flow.
• Furthermore, we can use this function to build Mish or other activation functions

Self evaluation:

1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: heka1024 heka1024@gmail.com

[taos-ci]

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2545. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

[taos-ci]

:octocat: cibot: @heka1024, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nntrainer/ci/repo-workers/pr-checker/2545-202404150909200.1524920463562-f83a0b89ae343d65d820be010f18fe5dcc54599a/.

[myungjoo]

Please rebase:

$ git checkout main
$ git pull ${FILL_upstream_name} main
$ git checkout activation-softplus
$ git rebase main
$ git push ${FILL_your_fork_name} HEAD -f

[skykongkong8]

I found this PR raises error when building with meson build -Denable-fp16=true @heka1024 Could you please take a look?

[heka1024]

@skykongkong8 I think precision could be a problem. Could you give me a link of failure log?

[skykongkong8]

@heka1024 Nope it's build fail, not unittest fail.

 error: converting to ‘_Float16’ from ‘float’ with greater conversion rank [-Werror]
  364 |     return static_cast<T>(log(1 + exp_util<T>(beta * x)) / beta);

 error: converting to ‘_Float16’ from ‘float’ with greater conversion rank [-Werror]
  374 |     return sigmoid<T>(beta * x);
      |            ~~~~~~~~~~^~~~~~~~~~
../nntrainer/layers/acti_func.h:269:52: note:   initializing argument 1 of ‘static T nntrainer::ActiFunc::sigmoid(T) [with T = _Float16]’
  269 |   template <typename T = float> static T sigmoid(T x) {
      |  

Try below to generate the error above:

cd ~/nntrainer/
meson build -Denable-fp16=true
cd ./build
ninja

Think your commit needs some fixes. Could you create a new PR to fix this issue?

[skykongkong8]

@heka1024 This issue is resolved via #2557 I would like to personally recommend you to check for fp16 build case as well next time!