When do you drop a term?

[hguan6]

I understand that the log-saturation function and regularization loss suppress the weights of the frequent terms. But when do you drop a term (setting the term weight to zero)? Is it when the logit is less or equal to zero, so that the log(1+ReLu(.)) function outputs zero?

[sclincha]

HI @hguan6 , The terms that have a weight of zero are not considered. More precisely, the term weights that do not pass the Relu ( i.e Relu(w)=0) remains zero after the log and so we discard them.

Best

[hguan6]

@sclincha Thank you!