Issues with Flattening Before Global Mean Pooling and Missing Non-Linearities Before Linear Layers

[gpefanis]

Hi! I've been working with your GCN implementation and noticed a few issues that could affect the model's performance:

1. Global Mean Pooling issue:

• In the current implementation, the tensor is flattened before applying tg.nn.global_mean_pool (in the forward pass). Flattening the tensor collapses the dimensions to: [batch size, resolution(num of regions)*features] so the pooling will be ineffective.

• This can be seen also in the input dimension of the first linear layer when creating the gcn model. In your example: (fc1): Linear(in_features=6750, out_features=256, bias=True), where 6750 = 675 regions * 10 features, so no reduction in data is applied between the last Chebconv and the first Linear, so I guess no pooling?

  GCN(
  (conv1): ChebConv(1, 32, K=2, normalization=sym)
  (conv2): ChebConv(32, 32, K=2, normalization=sym)
  (conv3): ChebConv(32, 10, K=2, normalization=sym)
  (fc1): Linear(in_features=6750, out_features=256, bias=True)
  (fc2): Linear(in_features=256, out_features=128, bias=True)
  (fc3): Linear(in_features=128, out_features=9, bias=True)
  (dropout): Dropout(p=0.2, inplace=False)
  )

2. Missing Non-Linearity Before Linear Layers:

• In the fully connected layers (fc1, fc2, fc3), there is no non-linearity introduced between the layers. This reduces the effectiveness of using three layers, as each layer is simply performing a linear transformation without any non-linear interaction. Addition of ReLU between each fully connected layer could make the model more expressive and improve performance.

x = self.fc1(x)
x = self.dropout(x)
x = self.fc2(x)
x = self.dropout(x)
x = self.fc3(x)

Suggested fix for FC layers
x = F.relu(self.fc1(x))
x = self.dropout(x)
x = F.relu(self.fc2(x))
x = self.dropout(x)
x = self.fc3(x)`

Would love to hear your thoughts and see if these can be addressed. Thanks for the great tutorial!

[PeerHerholz]

Hi @gpefanis,

thanks for bringing this up. We'll look into this.

Cheers, Peer