Create Transformer model

[Kairixir]

I've found a parallelisation issue. So far I found this issue where a possible solution offers changing the Positional encoding of Transformer.

I think I could get into the guts of Darts and change the positional encoding by creating a whole new custom encoder and decoder. Would be nice to have the Darts implementation though. Maybe I can find it in their GitHub.

Another approach can be to find different way of parallelisation. I found this article which shows how to parallelise huggingface implementation of transformer using Ray library

P.S. Playing with the Trainer in Darts Transformer class definition did not help. I guess it was tested only on GPUs

[Kairixir]

Can be worth to instantiate learning rate decay to improve loss at later part of learning (logic behing it)

[Kairixir]

Could be worth to implement early stopping so the model is not overfit (also decreases number of epochs, yay!) Details for this can be found at Darts Transformer model page at argument "pl_trainer_kwargs"

[Kairixir]

Guide for optimisation tuning in PyTorch