I have a questions regarding nn.Embedding layer. It is a layer that mimics F.one_hot + nn.Linear(..., bias=False) and implemented as a lookup table, a magnitude faster than one hot + linear combination.
By default, nn.Embedding is initialized as a normal distribution (mean 0, std 1). However, first layer of SIREN expects uniform distributed input at interval [-1, 1].
Should I initialize nn.Embedding as embedding.weight.uniform_(-1, 1) to match expectations of SIREN for input distributions?
Can I use nn.Embedding as a first layer of SIREN, and initialize it as proposed - embedding.weight.uniform_(-1 / in_features, 1 / in_features) - to get rid of two linear layers without non-linearity in between?
I have a questions regarding
nn.Embeddinglayer. It is a layer that mimicsF.one_hot + nn.Linear(..., bias=False)and implemented as a lookup table, a magnitude faster than one hot + linear combination.By default,
nn.Embeddingis initialized as a normal distribution (mean 0, std 1). However, first layer of SIREN expects uniform distributed input at interval [-1, 1].nn.Embeddingasembedding.weight.uniform_(-1, 1)to match expectations of SIREN for input distributions?nn.Embeddingas a first layer of SIREN, and initialize it as proposed -embedding.weight.uniform_(-1 / in_features, 1 / in_features)- to get rid of two linear layers without non-linearity in between?