The current method of calculating takes the average of the case-by-case reconstruction score.
This seems wrong because it divides by the difference between the zero-ablation loss and the source loss which in some cases will be very close to zero, especially with models with more layers, and these cases will get large weight. Taking the average difference in loss seems more principled to me.
PR also changes the corresponding test.
Would like to test it to check that it reduces the noise but getting annoying wandb errors "message":"Invalid sweep config: invalid hyperparameter configuration: source_data" at the moment.
The current method of calculating takes the average of the case-by-case reconstruction score.
This seems wrong because it divides by the difference between the zero-ablation loss and the source loss which in some cases will be very close to zero, especially with models with more layers, and these cases will get large weight. Taking the average difference in loss seems more principled to me.
PR also changes the corresponding test.
Would like to test it to check that it reduces the noise but getting annoying wandb errors
"message":"Invalid sweep config: invalid hyperparameter configuration: source_data"at the moment.