[Feature request] Add support for times series forecasting models and an example notebook

[gariepyalex]

TL;DR

• It is hard to encode/decode batches of time series in MLServer
• An idiomatic example would be helpful
• An new content type and/or inference runtime could help even more

Description

Forecasting models are nowadays a big area in ML, with models such a Facebook Prophet. Right now, it is not straightforward to serve time series forecasting models with MLServer. It would be very useful for the community to have an idiomatic example to follow. Moreover, I believe support for time series would be further improved by adding a new content-type specifically designed for time series.

Data representation

Time series present additional challenges mostly due to the data representation. A time series is represented by a list of timestamps and values of arbitrary length. Problems arises when adding support for adaptive batching (receiving multiple time series in the same call). Since each time series in the batch can have a different length, they do not fit nicely in tabular format (2D array) without complex padding.

Possible solutions

BYTES array inputs

One solution is to encode each time series as a JSON string (using the "BYTES" datatype, like in this examples https://github.com/SeldonIO/MLServer/tree/master/docs/examples/custom-json#send-test-inference-request-rest). While this works, it adds complexity to the payload. The callers to encode each time series as a JSON string, then wrap these strings into a valid Inference V2 payload. Also, the schema validation is weak, as the inner JSON string for the time series is not validated. It is up to the application code to do the validation.

    "inputs": [
        {
            "name": "time_series",
            "datatype": "BYTES",
            "shape": [-1]
        }
    ],

Concatenate time series

It is also possible to concatenate time series into a 1D array:

    "inputs": [
        {
            "name": "dates",
            "datatype": "BYTES",
            "parameters": {
                "content_type": "datetime"
            },
            "shape": [-1]
        },
        {
            "name": "values",
            "datatype": "FP64",
            "parameters": {
                "content_type": "np"
            },
            "shape": [-1]
        },
        {
            "name": "id",
            "datatype": "INT32",
            "parameters": {
                "content_type": "np"
            },
            "shape": [-1]
        }
    ],

Note that a third array is necessary to distinguish which indices in dates and values are part of each time series. Without it, there are a lot of corner cases with time series containing missing values, overlapping time series, unordered time series, etc.

While this is another solution to accept multiple time series, it still presents issues:

• The number of time series received and returned is implicit, you have to look at values in the id array.
• We send a lot of duplicated values for the ids

What would be an ideal solution?

Since the V2 Inference protocol only supports some basic data types, a new content type and codec specifically designed for time series could solve most issues mentioned above. Ideally, we should be able to decode a type List[pd.Series] directly from an inference request payload.

Going even further, a Prophet inference runtime would also be useful.

Additional context

This was discussed on Seldon's Slack a while back: https://seldondev.slack.com/archives/C03DQFTFXMX/p1654638409539339

[adriangonz]

Hey @gariepyalex ,

Thanks a lot for raising this one! As an OSS maintainer, it's great to see so well-formatted issues providing so much detail. :+1:

In order to help us understand more about what would be needed, would you be able to provide a small example at the prophet level showing what type of inputs does the model expect?

[gariepyalex]

Prophet models accept the following format:

The input to Prophet is always a dataframe with two columns: ds and y. The ds (datestamp) column should be of a format expected by Pandas, ideally YYYY-MM-DD for a date or YYYY-MM-DD HH:MM:SS for a timestamp. The y column must be numeric, and represents the measurement we wish to forecast.

(source)

Note that Prophet only accepts a single time series as an input. However, there exists other families of models where multiple time series are needed as an input (see [Hierarchical and grouped time series](Hierarchical and grouped time series)).

[adriangonz]

Thanks for that extra context @gariepyalex.

In that case, wouldn't it be enough to encode both columns as separate inputs and using the pd content type?

As in,

{
    "content_type": "pd",
    "inputs": [
        {
            "name": "ds",
            "datatype": "BYTES",
            "parameters": {
                "content_type": "datetime"
            },
            "data": ["2020-01-02", ... ],
            "shape": [-1]
        },
        {
            "name": "y",
            "datatype": "FP64",
            "parameters": {
                "content_type": "np"
            },
            "data": [1.3, ... ],
            "shape": [-1]
        }
    ]
}

[gariepyalex]

To accept multiple time series as an input, this would require to add an id column to distinguish between the series. It is discussed this possible solution in Concatenate time series in the original message.

Note that a third array is necessary to distinguish which indices in dates and values are part of each time series. Without it, there are a lot of corner cases with time series containing missing values, overlapping time series, unordered time series, etc. While this is another solution to accept multiple time series, it still presents issues:

• The number of time series received and returned is implicit, you have to look at values in the id array.
• We send a lot of duplicated values for the ids

This is also incompatible with adaptive batching.

[adriangonz]

Thanks for clarifying that point @gariepyalex .

In that case, would it make sense to add an extra dimension to the y and ds inputs above? This would represent the "batch dimension", which is not too different from what other models do.

Although, based on your previous comments, would prophet understand that? I guess probably not... Maybe we'll need a new prophet runtime? I'm also not sure how well would the Pandas codec work with that...

[gariepyalex]

In that case, would it make sense to add an extra dimension to the y and ds inputs above?

Adding a dimension would require all time series to have the same length, which would be a large limitation.

[adriangonz]

Right, I think I'm starting to understand the problem now.

Based on your ideal solution, we would need to find a way to pass variable-length lists as the inputs of y and ds, and then somehow identify the different segments.

I agree that passing the id list is not ideal though. Instead, we could pass an extra parameter which contains the lengths of each segment? So, to represent a payload with 3 time series (with lengths 2, 3 and 4), we could do something along these lines:

{
    "content_type": "pd",
    "parameters": {
       "lengths": [2, 3, 4]
    },
    "inputs": [
        {
            "name": "ds",
            "datatype": "BYTES",
            "parameters": {
                "content_type": "datetime"
            },
            "data": ["2020-01-02", ... ],
            "shape": [9]
        },
        {
            "name": "y",
            "datatype": "FP64",
            "parameters": {
                "content_type": "np"
            },
            "data": [1.3, ... ],
            "shape": [9]
        }
    ]
}

[gariepyalex]

Great suggestion! Is parameters compatible with adaptive batching? Could two payloads

"parameters": {
       "lengths": [2, 2]
    },

and

"parameters": {
       "lengths": [3, 3]
    },

be merged into a single payload?

"parameters": {
       "lengths": [2, 2, 3, 3]
    },

I don't need adaptive batching for now, but I could easily imagine using for instance LSTMs where batching is required. Having the same payload format for any time series algorithm would be great.

[adriangonz]

That's a great point @gariepyalex! That probably wouldn't work with adaptive batching...

In that case, we could just include a new parameter that represents the lengths (i.e. moving the lengths parameter to its own separate entry in the inputs). Although we would also need to ensure that adaptive batching works with input heads of variable length between them (i.e. having 8 elements in y and 2 in lengths), which is not supported currently. It shouldn't be too tricky to add though (famous last words!)...

[mattfraz]

Checking in on this open issue. I'm wondering if there are any plans on the roadmap to implement support for a Prophet inference runtime?