search

XuGW-Kevin / DrM

DrM, a visual RL algorithm, minimizes the dormant ratio to guide exploration-exploitation trade-offs, achieving significant improvements in sample efficiency and asymptotic performance across diverse domains.
MIT License
53 stars 8 forks source link

Computing the dormant neurons values #4

Closed pseudo-rnd-thoughts closed 3 weeks ago

pseudo-rnd-thoughts commented 3 weeks ago

Thanks for open sourcing your work

I was looking through your code, in particular, the cal_dormant_ratio function. You compute the absolute value of the forward pass of observations, why? As you are using ReLU with your linear layers then for negative values, these have an activation of 0 not positive given abs

Was this intentional?

If not, this would implement that a larger percentage of your neurons are dormant than the paper suggests

XuGW-Kevin commented 3 weeks ago

Hi, thanks for your question! Actually, we used the definition in The Dormant Neuron Phenomenon in Deep Reinforcement Learning, which is the paper that "designed" dormant ratio. I agree that there might be a larger percentage of neurons are dormant than the paper suggests using this definition. Therefore, in the follow-up work ACE, we changed the definition to "gradient", which might solve this issue.

pseudo-rnd-thoughts commented 3 weeks ago

Thanks, I hadn't spotted that. Good luck with the paper