Has anybody used the h-1transformer-1d model in sequence classification? There is a closed issue, in which @lucidrains mentioned two approaches. I see those two models seem to both use separate Transformer to do classification. Is it a simple way to classify a sequence? For example, in the given example, we got a tensor with shape of (1, 8000, 512) after x = self.layers(x, mask = mask), can we use a simple pooling layer followed by a classification head? If so, how should we do this pooling?
Has anybody used the h-1transformer-1d model in sequence classification? There is a closed issue, in which @lucidrains mentioned two approaches. I see those two models seem to both use separate Transformer to do classification. Is it a simple way to classify a sequence? For example, in the given example, we got a tensor with shape of (1, 8000, 512) after
x = self.layers(x, mask = mask), can we use a simple pooling layer followed by a classification head? If so, how should we do this pooling?Any suggestions are highly appreciated.