search

dllllb / pytorch-lifestream

A library built upon PyTorch for building embeddings on discrete event sequences using self-supervision
Apache License 2.0
215 stars 46 forks source link

Batchnorm in NN Head #157

Open ivan-chai opened 4 months ago

ivan-chai commented 4 months ago

There are two problems with nn.Head:

  1. Linear layer when followed by BatchNorm must not include a trainable bias vector (bias=False).
  2. Batchnorm needs a tensor with shape (B, C, L), while PTLS uses (B, L, C). The conversion between these two formats must be added to the forward method.