search

zseder / hunvec

Sequential Tagging in NLP using neural networks
5 stars 4 forks source link

better log-sum-exp approximation #52

Closed pajkossy closed 9 years ago

pajkossy commented 9 years ago

log(sum(exp(z{i}))) = max{i}z{i} + log(sum(z{i} - max{i}z_{i}))

zseder commented 9 years ago

I implemented it, but we'll have to cite it somehow in the code. Where does it come from?

zseder commented 9 years ago

isn't there an exp() somewhere in the equation? this yields to log(0) by the way, we have to verify the equation by its source.

pajkossy commented 9 years ago

well, you're right, it should be max{i}z{i} + log(exp(sum(z{i} - max{i}z{i}))) (and it is not just an approximation) I took it from Hugo Larochelle's slide (University of Sherkbrooke, Neural Networks course), but I guess its usage is so common, that we don't have to cite it

zseder commented 9 years ago

done in final commit https://github.com/zseder/hunvec/commit/8d0690e44429a3fcebcfed8dba167bbfcf33103f