Transfer Entropy on Different Dimensions?

[xanderdunn]

The README mentions that transfer entropy can be calculated using the partial mutual information. If we are looking for the transfer from X->Y, I believe this would be the calculation:

import numpy as np
from entropy_estimators import continuous

np.random.seed(42)
x = np.random.rand(3000)
y = np.random.rand(3000)

# T(X->Y) = PMI(Y_future, X_past, Y_past)
transferXtoY = continuous.get_pmi(y[-3000:], x, y[0:3000], estimator="fp")

print(transferXtoY)

This produces the output value -1.0. The values are random so no transfer is expected.

Typically, I believe transfer entropy is calculated as a measure of entropy from past values of X to future values of Y given past values of Y. The requirement that parameters x, y, and z in get_pmi all have the same dimension seems to break the usefulness of this? For example, calculating the transfer on the most recent 200 values of Y given the history of X and Y does not work:

transferXtoY = continuous.get_pmi(y[-200:], x, y[0:3000], estimator="fp")

output:

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    transferXtoY = continuous.get_pmi(y[-200:], x, y[0:3000], estimator="fp")
  File "/usr/local/anaconda3/envs/plutus/lib/python3.7/site-packages/entropy_estimators/continuous.py", line 54, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/anaconda3/envs/plutus/lib/python3.7/site-packages/entropy_estimators/continuous.py", line 372, in get_pmi
    xz  = np.c_[x,z]
  File "/usr/local/anaconda3/envs/plutus/lib/python3.7/site-packages/numpy/lib/index_tricks.py", line 406, in __getitem__
    res = self.concatenate(tuple(objs), axis=axis)
  File "<__array_function__ internals>", line 6, in concatenate
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 200 and the array at index 1 has size 3000

Although y and z are the same dimension, x is not.

Is this the usage of transfer entropy you had in mind, or a different use case? Appreciate any ideas on usage.

[paulbrodersen]

When computing the transfer entropy TE(X->Y), you typically only consider the past states of X and Y at a single lag \tau, with \tau typically being 1. Your first example would then be:

import numpy as np
from entropy_estimators import continuous

np.random.seed(42)
x = np.random.rand(3000)
y = np.random.rand(3000)

# T(X->Y) = PMI(Y_future, X_past, Y_past)
transferXtoY = continuous.get_pmi(y[1:], x[:-1], y[:-1], estimator="fp")

print(transferXtoY)

I think this is what you meant to write in your first example but you didn't quite get the offsets right, as

print(np.all(y[-3000:] == y[0:3000]))
# True

You are completely right that it often would be useful to consider more than a single lag value. If you wanted to condition on the last two values of Y, the code would look like this:

z = np.c_[y[:-2], y[1:-1]]
print(z.shape)
# (2998, 2)
transferXtoY = continuous.get_pmi(y[2:], x[1:-1], z, estimator="fp")

However, just because you can, doesn't mean you should. Note that you have increased the dimensionality of the joint probabilities that (implicitly) have to be estimated. Each additional dimension exponentially increases the number of data points needed to get a good estimate of the local density. This is analogous to the fact that the number of data points to construct a decent histogram increases exponentially with the dimensionality of the data: for a 1D histogram you can maybe get away with 30+ points whereas for a 2D histogram you probably need 1000+, i.e. 30^2 points.

There are alternative approaches that are often more fruitful. The simplest option, in my opinion, is to impose a model on how past states of Y influence future states of Y. For example, instead of conditioning on the last 10 values of Y (thus adding 10 dimensions), you could condition on the (weighted) sum of the last 10 values (which would again just be a 1D vector).

Having such a condensed representation of a variables history is also preferable for another subtle but important reason: the k-NN estimators are not scale invariant. As a consequence, with the naive approach

z = np.c_[y[:-2], y[1:-1]]
print(z.shape)
# (2998, 2)
transferXtoY = continuous.get_pmi(y[2:], x[1:-1], z, estimator="fp")

you are effectively assuming that states at time t=-2 have an equally strong predictive power on current states as do states at t=-1, whereas in reality, states at t=-1 are often more predictive of the current state than previous states.
Having a more explicit forward model allows you to bake better assumptions into your calculations, but without knowing what data you are working with, it is pretty difficult to advise you on that front.

[xanderdunn]

Ah, I was missing the lag, thank you!

The note on the data requirements for accurate approximation as a function of the lag is very important, thank you for mentioning. A 1D vector of the weighted sum of the past n values is a great idea for an efficient approximation.

This clarifies the usage for me and is in line with my theoretical understanding of transfer entropy, I'll go ahead and close.