IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2) when doing keys.transpose

[mahdiabdollahpour]

I run the code using the examples but I get this error on net(x), when model is doing keys.transpose in multiHeadAttention.py about line 91 in

# Scaled Dot Product
        self._scores = torch.bmm(queries, keys.transpose(1, 2)) / np.sqrt(K)

the error is IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2) actually, keys must be in shape (batch_size, K, d_model), but it is not, so it has 2 dim instead of 3, so there is this error.

but why keys tensor is not (batch_size, K, d_model)?

code to produce the error:

import numpy as np
from matplotlib import pyplot as plt
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from tqdm import tqdm
import seaborn as sns;

sns.set()
from tst import Transformer

BATCH_SIZE = 4
NUM_WORKERS = 0
LR = 1e-2
EPOCHS = 2
attention_size = 24  # Attention window size
dropout = 0.2  # Dropout rate
pe = None  # Positional encoding
chunk_mode = None

K = 300  # Time window length
d_model = 48  # Lattent dim
q = 8  # Query size
v = 8  # Value size
h = 4  # Number of heads
N = 4  # Number of encoder and decoder to stack
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(f"Using device {device}")

d_input = 11  # From dataset
d_output = 1  # From dataset
from excel_to_oze import getdata
data = np.random.random_sample((300,12))
# data = getdata()
data = data.astype(np.float32)
# data = torch.from_numpy(data,device=device)
data = torch.tensor(data, device=device)
print(data.shape)
dataloader = DataLoader(data,
                        batch_size=BATCH_SIZE,
                        shuffle=True,
                        num_workers=NUM_WORKERS
                        )

# Load transformer with Adam optimizer and MSE loss function
net = Transformer(d_input, d_model, d_output, q, v, h, N, attention_size=attention_size, dropout=dropout,
                  chunk_mode=chunk_mode, pe=pe).to(device)

optimizer = optim.Adam(net.parameters(), lr=LR)
loss_function = nn.MSELoss()

# Prepare loss history
hist_loss = np.zeros(EPOCHS)
for idx_epoch in range(EPOCHS):
    running_loss = 0
    with tqdm(total=len(dataloader.dataset), desc=f"[Epoch {idx_epoch+1:3d}/{EPOCHS}]") as pbar:
        for idx_batch, batch in enumerate(dataloader):
            optimizer.zero_grad()
            x = batch[:, :-1]
            y = batch[:, -1]
            print(x)
            # print(y)
            # Propagate input
            netout = net(x)

            # Comupte loss
            loss = loss_function(netout, y)

            # Backpropage loss
            loss.backward()

            # Update weights
            optimizer.step()

            running_loss += loss.item()
            pbar.set_postfix({'loss': running_loss / (idx_batch + 1)})
            pbar.update(BATCH_SIZE)

    hist_loss[idx_epoch] = running_loss / len(dataloader)
plt.plot(hist_loss, 'o-')
print(f"Loss: {float(hist_loss[-1]):5f}")

version 3 python 3.7.6 torch '1.5.0+cu101' windows 10

[rotameraklisi]

Hi mahdiabdollahpour

is the problem solved?

[mahdiabdollahpour]

Hi mahdiabdollahpour

is the problem solved?

Hi

It's solved, but I do not remember exactly how?:) I think my input data dimension was not correct.

[rotameraklisi]

Got it, I'm having a similar problem, can you help? Code: [https://stackoverflow.com/questions/65459540/indexerror-dimension-out-of-range-expected-to-be-in-range-of-2-1-but-got]

[mahdiabdollahpour]

I'll check it out. In the meantime, here is the code that was working for me. https://github.com/mahdiabdollahpour/Time-Series-with-Transformers-for-Brent-Oil-Price/blob/master/main.py

On Sun, Dec 27, 2020 at 12:34 AM Nur notifications@github.com wrote:

Got it, I'm having a similar problem, can you help? Code: [ https://stackoverflow.com/questions/65459540/indexerror-dimension-out-of-range-expected-to-be-in-range-of-2-1-but-got ]

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/maxjcohen/transformer/issues/21#issuecomment-751395989, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGOPPLO33BHD2Z7RTBZJXX3SWZFVLANCNFSM4SH6EXBQ .

--

Mohammad Mahdi Abdollah Pour

*Github https://mahdiabdollahpour.github.io/, *LinkedIn https://www.linkedin.com/in/mohammadmahdi-abdollahpour-a48b008b/