Closed hguan6 closed 2 years ago
HI @hguan6 , The terms that have a weight of zero are not considered. More precisely, the term weights that do not pass the Relu ( i.e Relu(w)=0) remains zero after the log and so we discard them.
Best
@sclincha Thank you!
I understand that the log-saturation function and regularization loss suppress the weights of the frequent terms. But when do you drop a term (setting the term weight to zero)? Is it when the logit is less or equal to zero, so that the log(1+ReLu(.)) function outputs zero?