According to the highlighted sentences in yellow from the attached paragraph, zero KLD indicates a constant output and hence collapse. However, I thought the zero value of KLD means two distributions (here, the teacher output distribution and the student one) become identical. I don't understand why two distributions becoming identical means a collapse and what a constant output exactly means. If someone gives me a hint, that would be a huge help! Thanks!
Hi,
According to the highlighted sentences in yellow from the attached paragraph, zero KLD indicates a constant output and hence collapse. However, I thought the zero value of KLD means two distributions (here, the teacher output distribution and the student one) become identical. I don't understand why two distributions becoming identical means a collapse and what a constant output exactly means. If someone gives me a hint, that would be a huge help! Thanks!