Add support for multivariate structural time series

[bgroenks96]

Are there any plans or existing prototypes for extending the TFP STS API to handle multivariate observations?

I must admit to being somewhat new to STS, but I don't see any obvious theoretical limitations. There seems to be existing literature on the subject.

I would also be interested in possibly contributing to this extension!

[brianwa84]

We don't have any near term plans to add support, but acknowledge it could be very useful. I believe the underlying tfp.distributions.LinearGaussianStateSpaceModel already supports multivariate observations, so it would be a matter of adding support throughout tfp.sts. This could be a little tricky, as the code is likely assuming scalar observations in many places.

[bgroenks96]

Yes, LinearGaussianStateSpaceModel already supports it; that's why I figured it wouldn't be (too) hard... but I'm sure there will be problems. The trick would be how to set up the observation matrix in the higher dimensional cases, I guess.

From an API standpoint, do you think it would make more sense to generalize the existing STS types to multiple dimensions or to create a new MultivariateStructuralTimeSeries class of types?

[brianwa84]

I'd defer to @davmre .

My hunch is that it might be easier to at least make a Multivariate* implementation class to correspond with each current scalar class e.g. tfp.sts.LinearRegression gets a MultivariateLinearRegression look-alike. As to how that's exposed to the user, perhaps we could retain existing naming but defer to the alternative subclass with __new__ handling a kwarg like multivariate_observations=True (tfp.sts.LinearRegression(..., multivariate_observations=True))?

There are likely some performance penalties we would not want to pay in the non-multivariate case (e.g. cholesky can be slower than sqrt, on a 1x1 matrix).

[davmre]

I think I agree with Brian that it might be cleaner to separate out multivariate components in the class hierarchy---that'd at least make the shape semantics easier to disambiguate. I think there's enough here that it'd be worth maybe doing some simple prototyping and/or writing up a few paragraphs about potential designs before diving into writing a ton of code. A couple of questions I'd suggest thinking about:

• Which components have natural multivariate generalizations, and what might those look like (mathematically speaking)? Are there other 'multivariate-native' components we might want to support? E.g., I could imagine we'd want something like vector autoregression. Of course we don't have to implement all possible components, but it'd be good to at least have a sense of the types of things the API might want to support.

• Probably the most common multivariate settings involve hierarchical modeling, i.e., you have a bunch of related series that may share one or more common latent component(s) in addition to whatever they have going on individually. For example, you have 50 time series for the 50 US states, and you believe that each series is the sum of some local state-level effects plus one or more national-level effects that are shared across all states. We'd probably want at least a potential API story for how to support this---probably some sort of extension or multivariate version of tfp.sts.Sum?

There are likely some performance penalties we would not want to pay in the non-multivariate case (e.g. cholesky can be slower than sqrt, on a 1x1 matrix).

I think this mostly comes up at the lower level in tfd.LinearGaussianStateSpaceModel, which can already use scalar fast paths when static shape information is available (e.g., here). So it might not be necessary to worry about these issues in higher API levels --- as long as we can expose a clean and modular API that implements make_state_space_model, LGSSM should be able to take care of scalar-specific optimizations.

[brianwa84]

Hierarchical reconciliation would be very cool. (Is that where you're going w/ the second bullet?)

On Tue, Oct 13, 2020 at 4:09 PM Dave Moore notifications@github.com wrote:

I think I agree with Brian that it might be cleaner to separate out multivariate components in the class hierarchy---that'd at least make the shape semantics easier to disambiguate. I think there's enough here that it'd be worth maybe doing some simple prototyping and/or writing up a few paragraphs about potential designs before diving into writing a ton of code. A couple of questions I'd suggest thinking about:

-

Which components have natural multivariate generalizations, and what might those look like (mathematically speaking)? Are there other 'multivariate-native' components we might want to support? E.g., I could imagine we'd want something like vector autoregression https://en.wikipedia.org/wiki/Vector_autoregression. Of course we don't have to implement all possible components, but it'd be good to at least have a sense of the types of things the API might want to support.

Probably the most common multivariate settings involve hierarchical modeling, i.e., you have a bunch of related series that may share one or more common latent component(s) in addition to whatever they have going on individually. For example, you have 50 time series for the 50 US states, and you believe that each series is the sum of some local state-level effects plus one or more national-level effects that are shared across all states. We'd probably want at least a potential API story for how to support this---probably some sort of extension or multivariate version of tfp.sts.Sum?

There are likely some performance penalties we would not want to pay in the non-multivariate case (e.g. cholesky can be slower than sqrt, on a 1x1 matrix).

I think this mostly comes up at the lower level in tfd.LinearGaussianStateSpaceModel, which can already use scalar fast paths when static shape information is available (e.g., here https://github.com/tensorflow/probability/blob/master/tensorflow_probability/python/distributions/linear_gaussian_ssm.py#L1736). So it might not be necessary to worry about these issues in higher API levels --- as long as we can expose a clean and modular API that implements make_state_space_model, LGSSM should be able to take care of scalar-specific optimizations.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tensorflow/probability/issues/1129#issuecomment-707979474, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFJFSI3FKBTFBXHU4HQ2I6TSKSXWXANCNFSM4SNMHYLQ .

[bgroenks96]

For example, you have 50 time series for the 50 US states

A little bit of election 2020 inspiration there @davmre ? ;)

This does sound cool, though. Do you have any related publications you can share? I'll have to read up on the details how such hierarchical models are actually constructed.

There is this 2018 paper on multivariate BSTS that might be a useful reference.