how to decide whether an operator is trainable

leovandriel / caffe2_cpp_tutorial

C++ transcripts of the Caffe2 Python tutorials and other C++ example code

BSD 2-Clause "Simplified" License

431 stars 94 forks source link

how to decide whether an operator is trainable #34

Open breadbread1984 opened 6 years ago

breadbread1984 commented 6 years ago

I am confusing about whether I shall put an operator into trainable_ops or non_trainable_ops. How do you decide it? I saw Scale operator was put in non_trainable_ops. Shall I put Pow in non_trainable_ops as well?

leovandriel commented 6 years ago

That's a good question. This trainable/non_trainable thing is a bit of a hack. It's mainly used to decide for which operators we need a to add a gradient in AddGradientOps. In general the rule is: if it's an operator that is a core part of the model output, then it's probably in trainable. I don't recall why Scale is not in trainable. Pow should be in trainable.