search

timeseriesAI / tsai

Time series Timeseries Deep Learning Machine Learning Python Pytorch fastai | State-of-the-art Deep Learning library for Time Series and Sequences in Pytorch / fastai
https://timeseriesai.github.io/tsai/
Apache License 2.0
5.15k stars 643 forks source link

Masking and Normalization order in PatchTST #734

Closed ikvision closed 1 year ago

ikvision commented 1 year ago

The current implementation of PatchTST is Normalization and then Masking: https://github.com/timeseriesAI/tsai/blob/3e481d501a81bf914644f64d51179d636d62caa9/tsai/models/PatchTST.py#L376-L379 I think this can lead to an unintentional leakage - the normalization takes into account the data from the patches that are masked. This order of Layers can leak information from the masked data during training and therefore the reconstruction task of the masked information is easier. I believe that masking and later normalizing the data during training can lead to better performance on the test set.

oguiza commented 1 year ago

Hi @ikvision, I'm not sure what masking you are referring to. AFAIK there's no masking in tsai`s PatchTST.

ikvision commented 1 year ago

@oguiza you are right, in the supervised patchTST there is no masking I was confused by the original paper self-supervised masking operation