Incorrect labels in demo.ipynb?

[noamwies]

The generate_data function shuffles the labels even though the predictions are ordered since they are multiplied by the inv_order matrix.

[neel04]

@yizhangzzz can you clarify this?

@noamwies I'm not sure what significance the bmm has, but I can confirm the labels returned by the data gen are shuffled - since they don't correspond to the variables, I don't think the results are reproducible..

[noamwies]

Hey @neel04, I was able to replicate the results after fixing the labels. However, I noticed that the results are highly dependent on the hyperparameters.

[neel04]

I still don't get you; if the permutation is random, then no baseline would actually be able to achieve a perfect score since the labels correspond the value of each of the variable. If those labels are scrambled, the model has actually no idea what labels correspond to what variables - it will simply just predict a random string of 0s and 1s.

Here's a sample from the datagen:

idx (2, 0, 3, 1)
sent  n = not d , z = val 1 ,  h = not n ,  d = val z , 
label [0, 1, 1, 1]
values [1 1 0 1]

idx is the new ordering which is applied to label to obtain values. And values are returned by the function. Note that label is the ground truth, where each index is for each var n, z, h, d respectively.

However, the datagen returns values which is clearly incorrect. So I'm honestly just confused here. Hopefully you have some idea what's going on here 😄

[KevinZhoutianyi]

@neel04 Hi Neel,

It seems that for the generated data, the labels that actually returned from the function in not permuted but the sent is permuted. Note that the premuted one is called label but labels is returned.

Hence, in the demo.ipynb, you need to use bmm to make the model's prediction in the correct order: