Parameters Initialization

[YoelShoshan]

I see that in multiple locations in the code where parameters are initialized you are using torch.empty, which looks like this: nn.Parameter(torch.empty( ...

I assume that this is to speedup model loading, since when used purely for inference it will be overridden by the checkpoint params anyway.

However, I'm trying to reproduce your training, how do you initialize the following within modules.py ?

Class Node2Edge

1. self.out_weights -
2. self.out_bias -

Class Attention: 1.self.qg_weights - 2 self.kv_weights - 3 self.qg_bias - 4 self.kv_bias -

5. self.o_weights -
6. self.o_bias -

class GeometricAttention:

1. self.linear_b_weights -
2. self.linear_b_bias -
3. self.act_w -
4. self.act_b -
5. self.out_proj_w -
6. self.out_proj_b -

Thanks!