Preprocessing for univariate/multivariate timeseries ?

[yanisIk]

Is your feature request related to a problem? Please describe. Complexity of preprocessing/transforming data for LSTM networks.

Describe the use case Most timeseries data come in the format where each row represents a timestamp or step. The generic preprocessing step for timeseries/sequences is to transform N rows to 1 row, using a sliding window for example.

Describe the solution you'd like A preprocessing feature that transforms N rows into one row, that supports sliding window or tumbling window and that supports univariate or multivariate timeseries/sequences.

Input:

• time, featureA, featureB, label
• 1, a, 1 X
• 2, b, 2 Y
• 3, c, 3 X
• 4, d, 4 Y

Params: timeColumn="time", N=3, WindowType="sliding" and label=["label"]

Output:

• featureA, featureB, label
• [a, b, c], [1, 2, 3] X
• [b, c, d], [2, 3, 4] Y

Describe alternatives you've considered Doing it by hand everytime.

I'm not very proficient in Python and manipulating matrices and multiple for loops can become confusing and very error prone. I'm looking for a simple script that can do that, even if it's not directly integrated as a preprocessor such as tokenizers.

Thanks

[w4nderlust]

This is a good idea. Ludwig's way of dealing with timeseries is reminiscent of text features and may be confusing to someone who is used to the timestamp based way to represent them. I will consider adding this in future releases.

[SirJohnFranklin]

It seem that doing it by hand is nasty work and it maybe is. Still, you can use add_sequence_feature_column() and you'll notice that it may take a lot of time (days in my case), depending on the size of your dataset. Thus, if the proposed method is not significantly faster it may not be an option do it at the creation of the model.

See add_sequence_feature_column(): https://uber.github.io/ludwig/examples/#time-series-forecasting-weather-data-example