How to reproduce Table 1?

[kainoj]

Hi! First of all, I am a big fan of your work, it's a really nice paper!

However, I am having a hard time reproducing the paper results in Table 1. Would you please share the command to debias, say DistilBERT on all-token mode?

I suppose

cd script
./preprocess.sh dbert ../data/news-commentary-v15.en   

goes first, and then? Does script/debias.sh define all required hyperparameters?

Also, could you confirm that in Table 1, reported SEATs are measured wrt the CLS token of the last layer?

[kanekomasahiro]

Thank you. Yes, that script is enough. However, the seed was not fixed, so the results may not be as in Table 1.

Also, could you confirm that in Table 1, reported SEATs are measured wrt the CLS token of the last layer?

Yes, I confirm that.

[kainoj]

Hi, thank you for a prompt answer!

I see that the flag: https://github.com/kanekomasahiro/context-debias/blob/7f01d08ce0f9c987f0ac9a45ec121bde1bdf4a5c/src/run_debias_mlm.py#L769 is not set in the script, so the loss can get negative: https://github.com/kanekomasahiro/context-debias/blob/7f01d08ce0f9c987f0ac9a45ec121bde1bdf4a5c/src/run_debias_mlm.py#L527-L528 which would mean that you effectively enforce the embeddings into opposite directions, as opposed to the orthogonality described in the paper.

Could you confirm this, please?

[kanekomasahiro]

Hi, thank you for pointing that out. I committed the change.

[kainoj]

Thanks, that explains quite a bit!