Separate Class / Script for Error Calculations

[alexlafleur]

1. matrix error
2. implied timescales
3. stationary distribution

These functions should be a parameter input for the error_bayes (or other) functions.

[greenTara]

Taking the current calculation as a model: np.linalg.norm(A0 - self.spectral_mm1_0_0_scaled.trans)

The new function is called "matrix_err"

has signature of A (an ndarray) and B (a Spectral_MM)

return np.linalg.norm(A - B.trans)

[greenTara]

For implied timescales

"timescale_mean_rel_err"

msm_estimated = pyemma.MSM(A) ts_actual = msm_estimated.timescales() ts_estimated = B.timescales() ts_rel_err[i] = mathf.abs(ts_actual[i] - ts_estimated[i])/ts_actual[i] # for i = 0, ..., nstates -1 return ts_rel_err.mean

For this to work, we need to make use of the timescales method on the super class (MSM) of Spectral_MM - I hope that works.

[greenTara]

For the stationary distribution

"stat_dist_vec_err"

msm_estimated = pyemma.MSM(A) pi_actual = msm_estimated.pi() pi_estimated = B.pi() pi_vec_err = np.linalg.norm(pi_actual - pi_estimated) return pi_vec_err

[greenTara]

A thought on these functions - instead of putting them into util, how about making them methods of Spectral_MM. Then instead of B, we have self.

[alexlafleur]

According to issue #35 we wanted to pass in the error function as a parameter to the evaluation script. If we have it a class method of spectral_MM we would first have to create an object of that class to get the error function. I think it's better to have it in a separate script from which we can choose the error function directly, don't you think?

[greenTara]

In that case, the function will take care of creating the object. I guess that is ok - I think we are only creating the object once in any case.

[alexlafleur]

The object is created within the evaluation script "error_bayes" or "error naive" by calling the eval function if I remember it right. So how should it know which error function of that object to use if we don't pass it in? We need something like a flag to tell which function to use, so it's easier to put in the function at first (but at the moment - if it's a function of spectralmm - we would have it earliest within the evaluation script)

[greenTara]

I don't know enough about how python works with functions to recommend a best way for this. At this point, whatever works is good.

[greenTara]

We don't have any unit tests for these new functions. Can we add tests to verify that the error is zero when the actual transition matrix is the same as the estimated one?

[alexlafleur]

For stat_dist_vec_err these lines cause troubles:

    pi_actual = msm_estimated.pi()
    pi_estimated = B.pi()

pi_actual = msm_estimated.pi() pi_estimated = B.pi() TypeError: 'NoneType' object is not callable

[greenTara]

The assertion in the unit test should be an "isClose" rather than equal to 0.

[greenTara]

I don't understand where the error regarding stat_dist_vec_err is arising.

[alexlafleur]

pi is not set. It is None in both msm_estimated and B

[greenTara]

This function is not implemented in the way I was expecting. My expectation was that if pi was not passed explicitly as a parameter, then it would be calculated.

But instead, .pi() is the parameter value that is passed, and _pi is the stationary distribution.

So I believe it should be:

pi_actual = msm_estimated._pi
pi_estimated = B._pi

[alexlafleur]

Yes, that solved it.