Open LiJiahao-Alex opened 2 months ago
Having the same issue, python version 3.11. I believe it has to do with the skip connections implemented in the Transformer class' forward() function. The first ParallelTransformerBlock block has a default dilation rate of 2, meaning it will produce half the output tokens (256 vs 512 in normal transformers in this example). You can check this using the other example using DilatedAttention. This also means you cannot add the skip-connection because now there is a discrepancy between the sequence dimension of the original input and the dilated output.
I'm not sure what the intention was of the original authors though.
Steps to reproduce:
python3.11 -m venv venv
and activate source ./venv/bin/activate
pip install longnet
import torch
from long_net.model import LongNetTransformer
longnet = LongNetTransformer(
num_tokens=20000,
dim=512,
depth=6,
dim_head=64,
heads=8,
ff_mult=4,
)
tokens = torch.randint(0, 20000, (1, 512))
logits = longnet(tokens)
print(logits)
Produces:
Traceback (most recent call last):
File "/Users/sander.moonemans/Study/Thesis/Longnet/longnet_transformer.py", line 14, in <module>
logits = longnet(tokens)
^^^^^^^^^^^^^^^
File "/Users/sander.moonemans/Study/Thesis/Longnet/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sander.moonemans/Study/Thesis/Longnet/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sander.moonemans/Study/Thesis/Longnet/venv/lib/python3.11/site-packages/long_net/model.py", line 302, in forward
x = self.transformer(x)
^^^^^^^^^^^^^^^^^^^
File "/Users/sander.moonemans/Study/Thesis/Longnet/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sander.moonemans/Study/Thesis/Longnet/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sander.moonemans/Study/Thesis/Longnet/venv/lib/python3.11/site-packages/long_net/model.py", line 271, in forward
x = block(x) + x
~~~~~~~~~^~~
RuntimeError: The size of tensor a (256) must match the size of tensor b (512) at non-singleton dimension 1
I ran the example program and got the following error.
It looks like there's something wrong internally?
Upvote & Fund