Training on Custom Dataset

[daksh-goyal]

Hey @chickenbestlover, I actually wanted to use your code on a different dataset that has 7 columns in total. The thing is I have around 100 sequence having around 1500 timestamps each. How do I split this data into training and testing so that I can train on your model. Furthermore, what changed would be required in the model architecture.

PS: I tried concatenating all 100 files and then dividing the total length in 80/20 ratio. I fed this to your model architecture without changing anything and got an error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-f9d48947f2f5> in <module>()
    222     for epoch in range(start_epoch, args.epochs+1):
    223       epoch_start_time = time.time()
--> 224       train(args,model,train_dataset,epoch)
    225       val_loss = evaluate(args,model,test_dataset)
    226       print('-' * 89)

5 frames
<ipython-input-8-f9d48947f2f5> in train(args, model, train_dataset, epoch)
    132       hids1 = []
    133       for i in range(inputSeq.size(0)):
--> 134         outVal, hidden_, hid = model.forward(outVal, hidden_,return_hiddens=True)
    135         outVals.append(outVal)
    136         hids1.append(hid)

/content/drive/My Drive/Colab Notebooks/YahooDataset/model/model.py in forward(self, input, hidden, return_hiddens, noise)
     60 
     61     #emb = self.layerNorm1(emb)
---> 62     output, hidden = self.rnn(emb, hidden)
     63     #output = self.layerNorm2(output)
     64 

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    565             hx = self.permute_hidden(hx, sorted_indices)
    566 
--> 567         self.check_forward_args(input, hx, batch_sizes)
    568         if batch_sizes is None:
    569             result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    521 
    522         self.check_hidden_size(hidden[0], expected_hidden_size,
--> 523                                'Expected hidden[0] size {}, got {}')
    524         self.check_hidden_size(hidden[1], expected_hidden_size,
    525                                'Expected hidden[1] size {}, got {}')

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in check_hidden_size(self, hx, expected_hidden_size, msg)
    185         # type: (Tensor, Tuple[int, int, int], str) -> None
    186         if hx.size() != expected_hidden_size:
--> 187             raise RuntimeError(msg.format(expected_hidden_size, tuple(hx.size())))
    188 
    189     def check_forward_args(self, input, hidden, batch_sizes):

RuntimeError: Expected hidden[0] size (2, 1837, 32), got (2, 64, 32)

Help would be appreciated. Thanks