API: Single Item Prediction

[jaheba]

Background

Prediction happens on an item to item bases. That means, during prediction all items are independent and thus we also forecast each item individually.

However, in some algorithms (NN based) we group items together during prediction for performance reasons.

Our predict-API always takes a whole Dataset and not an individual item.

This is a problem, since we loose a one-to-one relationship between input and output time-series.

My Problem

I'm currently thinking about a way to incorporate "fallback"-forecasts, where a model isn't able to produce a result and thus want to use another method to fall back on. Our current API design makes it difficult to handle the items individually, since I always have to handle a whole set of items.

What I want is to have something like this:

predictor = FallbackPredictor(
    PredictorA(...), # can't handle NaN values
    PredictorB(...)
)

Essentially, we would try A first on each item and fallback on B if no result could be produced on the first try.

Possible Solutions

At the moment I see two ways to solve this:

1. Split predictors in single-item and multi-item predictors with different behaviour and capabilities.

2. Require that all steps need to be capable of handling multiple items at once.

The obvious drawback of 1. being that not all predictors can be treated the same, and the drawback of 2. being the added complexity and hacks to make it work.

[vafl]

Yes. Essentially, the batching is a performance optimization (for throughput). The main benefit is for neural network models which can parallelize the batch.

For predictions behind an endpoint we usually use mxnet's naive engine, which does not parallelize. I don't know how much speedup the batching gives in that case.

We could measure this and if there is no big speedup we could switch to the single item predict method. Alternatively, could we maybe have an additional single item predict method or pass the batch size as a parameter when calling predict?

[lostella]

I think I like the idea (naming is arbitrary) of a predict_item predict_entry method implemented in specific predictors, and a default

def predict(self, dataset: Dataset) -> Generator[Forecast]:
    for item in dataset:
        yield self.predict_entry(item)

defined in Predictor.

How to do this in the GluonPredictor though? There we could have a predict_batch instead, a different implementation of predict that uses the InferenceDataLoader and calls predict_batch, and a predict_item predict_entry which maybe wraps the single item in a Dataset and calls predict?

[lostella]

I think #240 only partially solved this, and this API should be enforced (and tested) on all predictor types. Also, the naming there is not ideal in my opinion: if the method predicts a single DataEntry then it should probably be called predict_entry.