Closed koayon closed 3 weeks ago
I think this looks pretty reasonable. This:
total_entropy_F = bool_entropy_F.cuda() + bool_prob_F.cuda() * float_entropy
should be this, right?
total_entropy_F = bool_entropy_F.cuda() + bool_prob_F.cuda() * float_entropy_F
A runtime estimate + memory usage per SAE would be nice, and maybe we want to have an option to aggregate the MDL over multiple batches of activations?
I think this looks pretty reasonable. This:
total_entropy_F = bool_entropy_F.cuda() + bool_prob_F.cuda() * float_entropyshould be this, right?
total_entropy_F = bool_entropy_F.cuda() + bool_prob_F.cuda() * float_entropy_FA runtime estimate + memory usage per SAE would be nice, and maybe we want to have an option to aggregate the MDL over multiple batches of activations?