Sanity check failed

[veronica320]

Hi there,

Thanks for your awesome work and the nice repo! I'm running usage_example.ipynb, and everything is fine until the "sanity checks" code block. All three assertion statements failed initially:

def do_sanity_check(P, Ws, x_train):
    assert np.allclose(P.dot(P), P)
    assert np.allclose(Ws[0].dot(P.dot(x_train[0])), 0.0)

    for w in Ws:
        for w2 in Ws:
            if w is w2: continue
            assert np.allclose(w.dot(w2.T).item(), 0.0)

do_sanity_check(P, Ws, x_train)

The first two (assert np.allclose(P.dot(P), P) and assert np.allclose(Ws[0].dot(P.dot(x_train[0])), 0.0) are resolved once I set the tolerance threshold a little higher (atol=1e-07 and atol=1e-06 are enough respectively). But for the third assertion in the for loop (assert np.allclose(w.dot(w2.T).item(), 0.0)), the values are way off. I printed w.dot(w2.T).item() in the loop everytime:

-0.2324167937040329
-0.177420973777771
-0.2324167937040329
0.011475482024252415
-0.177420973777771
0.011475482024252415

The values are quite far from 0.

Do you know what might be going wrong? Does this mean the classifiers aren't pairwise orthogonal & what's the significance of it?

Thanks in advance!

[shauli-ravfogel]

Thanks for reaching out! Indeed, the classifiers seem to not be orthogonal, so the set of all K classifiers may actually not span a K-dimensional subspace (there may an "overlap" in the information they rely on).

I will try to understand what's going on there soon, but on the meantime, you can use the alternative script debias.py -- the one which used for the experiments in the paper. There, the orthogonality constraint is maintained as expected.

I will update once this is figured out.

Thanks,

Shauli

[veronica320]

Thanks for your quick reply! Much appreciated!

[zhiqihuang]

Hi,

Thanks for the code. I had the same issue. Do you have any ideas or updates about this?

Basically, debias.py is using sklearn while inlp.py is based on pytorch (lightning). The model and data are the same. Why do different training packages cause the problem?

I appreciate your help.