search

lgarithm / crystalnet

crystalnet -- a mini core AI library (being refactored, see https://github.com/lgarithm/stdnn-ops)
MIT License
16 stars 3 forks source link

Operator fusion #60

Closed lgarithm closed 5 years ago

lgarithm commented 6 years ago

Currently the operation

y = f(x) = a(wx + b)

where a is the activation function, requires 3 steps to compute, but we can combine them into 1 or 2 steps.

For training, we need two steps: f(x) = a(g(x)) where g is an affine operator.

For inference, we only need one step.

lgarithm commented 6 years ago

We need to introduce the concept of inplace operator. An operator can be used as inplace operator if the shape of its output tensors equals one of its input tensors.

lgarithm commented 6 years ago

This should be done when compiling a symbolic model to physical model.

lgarithm commented 5 years ago

supported by https://github.com/lgarithm/stdnn-ops/pull/1