search

microsoft / CNTK

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit
https://docs.microsoft.com/cognitive-toolkit/
Other
17.52k stars 4.28k forks source link

Loss Function with L2 Regularization #2427

Open ankur6ue opened 7 years ago

ankur6ue commented 7 years ago

Hello, I'm attempting to add L2 regularization on the network weights to the softmax with cross entropy loss function. Here's my implementation:

def cross_entropy_plus_softmax_with_regularization(model, labels, l2_regularization_weight): w_norm = C.Constant(0); for p in (model.parameters): w_norm = C.plus(w_norm, 0.5*C.reduce_mean(C.square(p)))

return C.reduce_log_sum_exp(model.output) - C.reduce_log_sum_exp(C.times_transpose(labels, model.output)) + l2_regularization_weight*w_norm

Does this look correct? It appears to work.. The whole symbol expression API is pretty wonky and takes some getting used to.

Thanks, Ankur

ankur6ue commented 7 years ago

one small typo: the reduce_mean should be a reduce_sum as L2 normalization doesn't take an average over the weights..

cha-zhang commented 7 years ago

For L2 regularization the learners support it natively. Check for example: https://cntk.ai/pythondocs/cntk.learners.html?highlight=regularization#cntk.learners.sgd