Non-probabilistic Version?

[sbonner0]

Hi,

Thanks for your work, I really like the code. I was wondering if you offered the Non-probabilistic version as discussed in the original paper? If not do you think it would be hard to add? I’m happy to give it a go with a few pointers.

Kind Regards,

[vmasrani]

I didn't, but it should be straightforward to implement. All you'll need is a single model similar to the GCNEncoder, except with 1) one GCN in the init rather than two, and 2) a modified forward method that reconstructs the adj matrix as specified in eq'n 4 in the paper. Then to train use a reweighted cross entropy loss from pytorch. For the deterministic case you won't need pyro or a separate encoder/decoder as in the probabilistic case.

[sbonner0]

That’s great, thank you. I’ll give it a go tomorrow. You essentially have eq’n 4 in the InnerProductDecoder function right?

[vmasrani]

Yup, exactly.

[sbonner0]

So I gave it a go - https://github.com/sbonner0/gae_in_pytorch I created a new file train_GAE.py, in which we train the non-probabilistic model.

However there is no 'weighted_cross_entropy_with_logits' equivalent in pytorch so I had to write one similar to https://github.com/pytorch/pytorch/issues/5660

I'm not able to get the same kind of results in the original paper though, would you mind having a look and seeing if I am missing anything obvious?

[vmasrani]

Hmm, I couldn't spot any bugs, but I haven't the time to look very closely. However I believe torch.nn.CrossEntropyLoss allows you to specify weights, so you may want to try using their loss function to see you get the same results.

[sbonner0]

Great, thank you for taking the time to look!

Regarding the loss - i thought that essentially we are doing a weighted binary cross entropy loss? Therefore nn.CrossEntropyLoss is not applicable. Also the weight parameter is for class weights i believe. But yes, not having an equivalent to tf.nn.weighted_cross_entropy_with_logits in pytorch is a pain! I believe a work around will be possible when pytorch 0.4.0 is realised.

[AshStuff]

@sbonner0 Why not just use binary_cross_entropy_with_logits

[sbonner0]

At the time, pytorch 0.3 didn’t have the option to pass in a weight to the cross entropy loss. Obviously it does now :)

[vmasrani]

Updated to use binary_cross_entropy_with_logits :)

[AllenWu18]

At the time, pytorch 0.3 didn’t have the option to pass in a weight to the cross entropy loss. Obviously it does now :)

Hi, I just read your code and compared it with the original code of vmasrani's, and I have some question: 1) why does it need bias in class GraphConvolution() in layers.py? In your code there is "bias=True" but in the original code it is "bias=False" 2)I don't understand the difference between function "reset_parameters" in your code and the original code of vmasrani's. Are they same in effect? Thx!

[sbonner0]

Are those questions directed at me @AllenWu18 ?

[AllenWu18]

Yes, I can’t find your mailbox address to ask these questions, so I just set them here. Can you answer them?Thx very much! ：）

	allenwu18 邮箱：allenwu18@163.com

Signature is customized by Netease Mail Master

On 04/16/2019 18:51, Stephen Bonner wrote:

Are those questions directed at me @AllenWu18 ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sbonner0]

Hi Allen,

1) - The bias flag just control if to use a bias in each GCN layer or not...for my application i found it to give a small lift in performance. Basically the bias changes the GCN layer wise propagation rule from ht = GCN(A, ht-1, W) to ht = GCN(A, ht-1, W + b). 2) - The reset parameters function just determines the initialization of the weight matrices. You could change this to whatever you wanted (xavier for example), but i just initialise from a scaled random uniform distribution.

Hope that helps!

[AllenWu18]

Very Thx！ It helps me :) And may I ask another question? In the VGAE original paper I remember the author say when use GAE, the reconstruction graph is computed by an activation function apply in the adj_hat( adj_hat = torch.mm(x,x.t()) So，should I add a ReLU like adj_hat = F.relu(adj_hat) in def forward of class GAE? I think this is same to the paper, but when I added the relu, the loss and other score to evaluate became wave..... So should we add the activation function or not?Or should it be sigmoid, instead of ReLU? Hope to your reply！

	allenwu18 邮箱：allenwu18@163.com

Signature is customized by Netease Mail Master

On 04/17/2019 00:17, Stephen Bonner wrote:

Hi Allen,

The bias flag just control if to use a bias in each GCN layer or not...for my application i found it to give a small lift in performance. Basically the bias changes the GCN layer wise propagation rule from ht = GCN(A, ht-1, W) to ht = GCN(A, ht-1, W + b). The reset parameters function just determines the initialization of the weight matrices. You could change this to whatever you wanted (xavier for example), but i just initialise from a scaled random uniform distribution.

Hope that helps!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sbonner0]

adj_hat will need to be passed through a sigmoid function if you want to use it for predictions, but the loss function used in pytorch already contains the sigmoid as part of the loss function.

[AllenWu18]

So does it means the loss function F.binary_cross_entropy_with_legits() has already contained the sigmoid?

	allenwu18 邮箱：allenwu18@163.com

Signature is customized by Netease Mail Master

On 04/17/2019 01:05, Stephen Bonner wrote:

adj_hat will need to be passed through a sigmoid function if you want to use it for predictions, but the loss function used in pytorch already contains the sigmoid as part of the loss function.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sbonner0]

Indeed it does: https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss

[AllenWu18]

It helps.Thank you very much!

[AllenWu18]

And I would like to ask another question: in class GAE the function encode_graph is : def encode_graph(self, x, adj):

    # Perform the encoding stage using a two layer GCN
    x = F.relu(self.gc1(x, adj))
    x = F.dropout(x, self.dropout, training=self.training)
    x = self.gc2(x,adj)

    return x

And I wondered the function change the value of x( which is the features matrix of the graph), and is that means the original features we just use in the epoch 1, and in the following epochs, the x values have been changed already?? It made me puzzled :( Maybe I haven't gotten the idea of the paper....

[sbonner0]

Well the original vertex features X are passed in once every forward pass of the the GAE and never replaced, so the X passed into encode_graph will always be the original vertex features as they are never over written.