model has not been estimated

[noeliaferruz]

Hi guys,

I have come across an error when plotting timescales a few times now, but I don't know what I'm doing wrong. The error is huge, it's attached. Apparently the last line read:

AttributeError: Model has not yet been estimated. Call estimate(X) or fit(X) first

I have however run many times model=model(dataTica). But it doesn't seem to be the reason. Any help appreciated!

Noelia

error_impliedtimescales.txt

[j3mdamas]

This is a PyEMMA error. What version of htmd and pyemma are you using? Make conda update htmd (which should also call for the necessary pyemma version 2.1). Check your pyemma version with conda search pyemma. The one with the * should be 2.1. If not, please tell us.

If this does not solve, please report.

[stefdoerr]

It won't solve it. Actually I think it's an error report of the new pyEMMA. I've seen it around the last few days but during long batches of work so I could not determine the reason. @noeliaferruz the alignment of text is broken in that file. Can you paste the next report here and keep the console on so that you can then get in debug mode for me and save some variables for Frank?

[stefdoerr]

ah no the text file is fine sorry. I will mention it to the pyEMMA guys.

[noeliaferruz]

Hi, I can forward the trajectories for this system as well. It is also a rather short ensemble, like 1us, and only a protein on its membrane. let me know

[stefdoerr]

Seems like they are already on it:

https://github.com/markovmodel/PyEMMA/issues/764

Don't know if Frank really needs the trajectory. I guess they have their own data.

[giadefa]

I think that conda update htmd does not update anything else.

On 20 April 2016 at 20:09, João M. Damas notifications@github.com wrote:

This is a PyEMMA error. What version of htmd and pyemma are you using? Make conda update htmd (which should also call for the necessary pyemma version 2.1). Check your pyemma version with conda search pyemma. The one with the * should be 2.1. If not, please tell us.

If this does not solve, please report.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/Acellera/htmd/issues/19#issuecomment-212539704

http://www.acellera.com

   <https://twitter.com/acellera>

https://www.youtube.com/user/acelleracom https://www.linkedin.com/company/2133167?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2133167%2Cidx%3A2-1-2%2CtarId%3A1448018583204%2Ctas%3Aacellera https://www.acellera.com/md-simulation-blog-news/ http://is.gd/1eXkbS

[j3mdamas]

Well, it should now. I faced an error this afternoon because I had a recent version of htmd but an older version of pyemma. I think Stefan corrected it.

[j3mdamas]

Frank Noé says that this issue should be solved markovmodel/PyEMMA#764. It's solved in markovmodel/PyEMMA#784, and seems like the changes are merged into the devel branch. @stefdoerr, do we wait for the fix to be in the next release version of PyEMMA or do we fetch the devel?

[stefdoerr]

We can fetch devel localy to check if they introduced new incompatibilities with our code but we cannot really distribute it (unless we start packaging our own pyemma version which will become a mess sooner or later imho). So the users will have to wait for their next release. Hopefully soon.

[j3mdamas]

I agree with you. I can fetch devel locally and check this.

[j3mdamas]

@noeliaferruz, can you give me the trajs that were giving you problems for me to test? Maybe also with a copy of the code that was failing. Thanks!

@stefdoerr: do I check for incompatibilities just by running the tests?

[stefdoerr]

Yes you can run the tests, but running the tutorials would not be bad either. But you might need to get the data since you are not in the lab network.

[j3mdamas]

Oh, right, the famous data :) maybe next week?

[stefdoerr]

It should be available for download in the tutorial pages so you can just download it from there.

[stefdoerr]

Just remember that you will need to modify the paths, and don't commit the modified paths afterwards

[j3mdamas]

Oh, right! the links. thanks. no no. I am going to test a fresh installation just for this, so I won't commit anything.

[noeliaferruz]

I cannot upload files greater than 10MB, sorry. It crashes with all the systems i've tried, you can try any dataset.

[stefdoerr]

Noelia are you talking about the new devel version of pyemma or the old one?

On Wed, Apr 27, 2016 at 8:10 PM, Noelia Ferruz notifications@github.com wrote:

I cannot upload files greater than 10MB, sorry. It crashes with all the systems i've tried, you can try any dataset.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Acellera/htmd/issues/19#issuecomment-215151573

[noeliaferruz]

I am talking about passing the trajs to @j3mdamas Oh if you mean my version I'm using 1.0.16

[j3mdamas]

and stefan was talking about it failing on all systems you tried, because it seems like it was specific when you reported it (I think)

[noeliaferruz]

I don't know if it's specific, it crashes in my two systems. I haven't updated conda since I reported this, did you fetch devel locally only or if I update would it work (by updating htmd or pyemma?)?

[j3mdamas]

No, I have fetched it manually. I haven't tested anything yet, as I would like a (non) working example. if it's not confidential material, can you use https://www.wetransfer.com/? with the code, please. thanks!

[stefdoerr]

By the way Noelia, I think limiting your timescales lagtimes should avoid the crash for the time being

[noeliaferruz]

Sent. Mh no I tried a couple of times and it didn't. I'll keep randomizing a vector of lag times, let's see. thanks

[stefdoerr]

oof ok, then Joao should definitely take a look at if the new version fixed the bug because they might have missed something

[noeliaferruz]

@j3mdamas, in this same set, if you try model.markovModel(300,5) there's this error as well appearing, maybe they are related: (Perhaps because the timescale is terrible at 30ns).

`---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-13-39d5b6f4fdbe> in <module>()
      1 model=Model(dataTica)
----> 2 model.markovModel(300,5)
      3 #model.plotTimescales(lags=list(range(100,5000,50)))

path/to/site/current/python/lib/python3.5/site-packages/htmd/model.py in markovModel(self, lag, macronum)
     67 
     68         self.lag = lag
---> 69         self.msm = msm.estimate_markov_model(self.data.St.tolist(), self.lag)
     70         self.P = self.msm.transition_matrix
     71         self.micro_ofcluster = -np.ones(self.data.K+1, dtype=int)

path/to/site/current/python/lib/python3.5/site-packages/pyEMMA-2.1.1-py3.5-linux-x86_64.egg/pyemma/msm/api.py in estimate_markov_model(dtrajs, lag, reversible, statdist, count_mode, sparse, connectivity, dt_traj, maxiter, maxerr)
    505                     maxerr=maxerr)
    506     # estimate and return
--> 507     return mlmsm.estimate(dtrajs)
    508 
    509 

path/to/site/current/python/lib/python3.5/site-packages/pyEMMA-2.1.1-py3.5-linux-x86_64.egg/pyemma/_base/estimator.py in estimate(self, X, **params)
    341         if params:
    342             self.set_params(**params)
--> 343         self._model = self._estimate(X)
    344         self._estimated = True
    345         return self

path/to/site/current/python/lib/python3.5/site-packages/pyEMMA-2.1.1-py3.5-linux-x86_64.egg/pyemma/msm/estimators/maximum_likelihood_msm.py in _estimate(self, dtrajs)
    268             P = msmest.transition_matrix(self._C_active, reversible=self.reversible,
    269                                          mu=statdist_active, maxiter=self.maxiter,
--> 270                                          maxerr=self.maxerr)
    271         elif self.connectivity == 'none':
    272             # reversible mode only possible if active set is connected

path/to/site/current/python/lib/python3.5/site-packages/msmtools/estimation/api.py in transition_matrix(C, reversible, mu, method, **kwargs)
    939         if mu is None:
    940             if sparse_computation:
--> 941                 T = sparse.mle_trev.mle_trev(C, **kwargs)
    942             else:
    943                 T = dense.mle_trev.mle_trev(C, **kwargs)

msmtools/estimation/sparse/mle_trev.pyx in msmtools.estimation.sparse.mle_trev.mle_trev (msmtools/estimation/sparse/mle_trev.c:2266)()

Exception: Stationary distribution contains entries smaller than 1e-15 during iteration

[stefdoerr]

This is an old issue I have already raised here: https://github.com/markovmodel/PyEMMA/issues/714

[j3mdamas]

Thanks for the data Noelia. Still downloading it, I'll report ASAP, as I am dealing with multiple issues at the same time.

[j3mdamas]

Just to keep you guys in the loop, I think the latest devel branch of PyEMMA requires thermotools, a separate package they develop. I saw this when I was importing the new PyEMMA, with the new PyEMMA/pyemma/thermo/estimators/TRAM_estimator.py requiring to import that module. I checked release 2.1.1, and it didn't have that file. Still, to make sure and not bother Stefan, I asked them: markovmodel/PyEMMA#787

I can still download thermotools and try to test their fix, but maybe we need to think about this thermotools issue.

[j3mdamas]

OK, a thermotools conda update solves that issue. Anyway, Martin confirmed that it is now a dependency. I am going to test their fix to Noelia's issue now.

[j3mdamas]

@noeliaferruz, you just sent me the filtered/. But the simlist needs the prod/. Can you share it with me? Thanks!

[noeliaferruz]

Hi! Can't you create a simlist with the filtered data? sims = simlist(glob('./filtered/*/'), './filtered/filtered.pdb')

[j3mdamas]

Yes, sorry. Still much to learn for me :)

[noeliaferruz]

No probl!

thanks, let me know

[j3mdamas]

@stefdoerr, is this OK to import the fresh up to date modules?

In [1]:
import sys
sys.path.insert(1, '/home/joao/maindisk/SANDBOX/issue19/htmd')
sys.path.insert(1, '/home/joao/maindisk/SANDBOX/issue19/PyEMMA')
In [2]:

from htmd import *
from PyEMMA import pyemma
from glob import glob
%matplotlib inline
htmd.config(viewer='ngl')
Videos from the HTMD2015 workshops are available on the Acellera youtube channel: https://www.youtube.com/user/acelleralive

You are on the latest HTMD version (unpackaged).
In [3]:

print(os.path.abspath(pyemma.__file__))
/data/joao/maindisk/SANDBOX/issue19/PyEMMA/pyemma/__init__.py

the final output says that I should using the right pyemma.

If this is OK, then I bump into an import error later on:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-6-62b623be30de> in <module>()
      2 data.dropTraj()
      3 data.fstep=0.1
----> 4 tica = TICA(data, 20)
      5 dataTica = tica.project(3)

/home/joao/maindisk/SANDBOX/issue19/htmd/htmd/projections/tica.py in __init__(self, data, lag)
     57             p.stop()
     58         else:
---> 59             self.tic = tica(data.dat.tolist(), lag=lag)
     60 
     61     def project(self, ndim=None):

/home/joao/maindisk/SANDBOX/issue19/PyEMMA/pyemma/coordinates/api.py in tica(data, lag, dim, var_cutoff, kinetic_map, stride, force_eigenvalues_le_one, mean, remove_mean)
   1126 
   1127     """
-> 1128     from pyemma.coordinates.transform.tica import TICA
   1129     if mean is not None:
   1130         import warnings

/home/joao/maindisk/SANDBOX/issue19/PyEMMA/pyemma/coordinates/transform/__init__.py in <module>()
     31 """
     32 
---> 33 from .pca import *
     34 from .tica import *

/home/joao/maindisk/SANDBOX/issue19/PyEMMA/pyemma/coordinates/transform/pca.py in <module>()
     28 from pyemma.util.reflection import get_default_args
     29 
---> 30 from pyemma.coordinates.estimators.covar.running_moments import running_covar
     31 import numpy as np
     32 from decorator import decorator

/home/joao/maindisk/SANDBOX/issue19/PyEMMA/pyemma/coordinates/estimators/covar/running_moments.py in <module>()
     19 import warnings
     20 import numpy as np
---> 21 from .moments import moments_XX, moments_XXXY, moments_block
     22 
     23 __author__ = 'noe'

/home/joao/maindisk/SANDBOX/issue19/PyEMMA/pyemma/coordinates/estimators/covar/moments.py in <module>()
     94 import math, sys, numbers
     95 import numpy as np
---> 96 from .covar_c import covartools
     97 
     98 

ImportError: cannot import name 'covartools'

This covartools is a folder with stuff inside. It's there, I don't understand why it isn't importing... Can you guide me so I can help Noelia? Thanks and I apologize for disturbing you :/

[stefdoerr]

It's C code. I am quite sure it needs to be compiled before using the new pyEMMA. If you know your way around conda environments, make a new one and then use python setup.py install from the main pyEMMA directory. This will install the new pyEMMA in the new conda environment. Then use that environment for testing (and you won't need the ugly sys commands at the beginning).

[stefdoerr]

Otherwise if you don't want to bother with conda environments like me, make a new conda installation and install the new pyEMMA in that folder. The first solution is more "correct" though I guess.

[j3mdamas]

Yeah, you were right, I had to setup it first. Stupid me.

Anyway, there's a simple alternative, that even Noelia can implement. Martin told me that in conda there is a pyemma-dev. So, easier than the conda environments, it's to install pyemma-dev: conda install pyemma-dev With that, all calls to pyemma point to the dev (to reverse, just uninstall pyemma-dev). I ran @noeliaferruz script with this, and it went good. Just with a lot of warning, including one that seems related with what Noelia reported in between: WARNING - Ignored error during estimation: Stationary distribution contains entries smaller than 1e-15 during iteration.

Anyway, Noelia, you can install pyemma-dev and do your analysis. I still need to run our tests with the pyemma-dev, and after it I'll close this issue

[j3mdamas]

All inline tests ran. All tutorials ran, in particular the relevant ones (ligand binding analysis, villin and CXCL12) Issue closed.

[stefdoerr]

Good job! Any mention when there will be a new pyemma release?

[marscher]

Am 04.05.2016 um 15:36 schrieb Stefan:

Good job! Any mention when there will be a new pyemma release?

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/Acellera/htmd/issues/19#issuecomment-216866910

maybe this afternoon ;)