Closed WeForgot closed 3 years ago
I hate leaving issues and then nearly immediately being like "oh wait..." Turns out I forgot that the the ViT library outputs in using the class tokens rather than full hidden reps. I switched to one that returned those and we are in business. Sorry about that, closing the issue.
Hi all,
I am sorry if this is a dumb question but I am running into an issue that I can't seem to solve. My model is a generative encoder/decoder model where the encoder is a vision transformer and the decoder is a routing transformer (this repo 😄 ). The output is continuous valued so I cannot use the autoregressive wrapper.
For the longest time I used this without passing in "recieves_context" which obviously was silly and kind of circumventing the whole idea of having the ViT head. When I used the flag though I get the error below.
I ran through the code and I can definitely see where it is breaking but I really have no idea where to even start with alleviating that. For what it is worth the dims of everything are consistent:
x = torch.Size([8, 230, 64]) context = torch.Size([8, 64]) input_mask = torch.Size([8, 230])
and parameters I am initializing with for the routing transformer are:
RoutingTransformer(dim = 64, depth = 2, max_seq_len = 256, heads = 16, ff_glu = True, use_scale_norm = True, causal = True, receives_context=True)
Once again, this is probably a large issue with me misunderstanding the code but it works with other transformer architectures and I am not sure where to go.
Cheers!