Bug in entropy calculation?

xjdr-alt / entropix

Entropy Based Sampling and Parallel CoT Decoding

Apache License 2.0

3.05k stars 311 forks source link

Bug in entropy calculation? #83

Open artus-LYTiQ opened 1 month ago

artus-LYTiQ commented 1 month ago

current_entropy = -jnp.sum(current_prob * jnp.log2(jnp.clip(current_prob, 1e-10, 1.0))) is operating on a single probability value. Shouldn't you calculate entropy over a distribution?

LeonEricsson commented 3 weeks ago

where is this code?