Closed gph82 closed 8 years ago
Ouch. That looks like being due to a recent change in the Estimator class, but this function should be definitely run in one of the tests, so I don't see how we could have overlooked this. We may need a patch.
I'll have a look. Did this work with 2.0.4?
Am 08/04/16 um 18:09 schrieb Guillermo Pérez-Hernández:
pyemma.version '2.1-9-g4936d0d' This happened:
its[pp]= pyemma.msm.timescales_hmsm([dtraj[::rs]for dtrajin cl[pp].dtrajs],4, lags=lags_fr[rs].astype(int),nits=5)
|AttributeError Traceback (most recent call last) /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self) 371 try: --> 372 return self._model 373 except AttributeError: AttributeError: 'MaximumLikelihoodHMSM' object has no attribute '_model' During handling of the above exception, another exception occurred: AttributeError Traceback (most recent call last)
in () 3 for pp in prot: 4 pp = pp[:2] ----> 5 its[pp] = pyemma.msm.timescales_hmsm([dtraj[::rs] for dtraj in cl[pp].dtrajs], 4, lags=lags_fr[rs].astype(int), nits=5) 6 7 /home/mi/gph82/programs/PyEmma/pyemma/msm/api.py in timescales_hmsm(dtrajs, nstates, lags, nits, reversible, stationary, connectivity, mincount_connectivity, separate, errors, nsamples, stride, n_jobs, show_progress) 848 itsobj = _ImpliedTimescales(estimator, lags=lags, nits=nits, n_jobs=n_jobs, 849 show_progress=show_progress) --> 850 itsobj.estimate(dtrajs) 851 return itsobj 852 /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate(self, X, **params) 341 if params: 342 self.set_params(**params) --> 343 self._model = self._estimate(X) 344 self._estimated = True 345 return self /home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/implied_timescales.py in _estimate(self, data) 146 self._models, self._estimators = estimate_param_scan(self.estimator, data, param_sets, failfast=False, 147 return_estimators=True, n_jobs=self.n_jobs, --> 148 progress_reporter=self) 149 150 ### PROCESS RESULTS /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate_param_scan(estimator, X, param_sets, evaluate, evaluate_args, failfast, return_estimators, n_jobs, progress_reporter) 296 297 # container for model or function evaluations --> 298 res = pool(task_iter) 299 300 if progress_reporter is not None: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __call__(self, iterable) 798 # was dispatched. In particular this covers the edge 799 # case of Parallel used with an exhausted iterator. --> 800 while self.dispatch_one_batch(iterator): 801 self._iterating = True 802 else: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in dispatch_one_batch(self, iterator) 656 return False 657 else: --> 658 self._dispatch(tasks) 659 return True 660 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in _dispatch(self, batch) 564 565 if self._pool is None: --> 566 job = ImmediateComputeBatch(batch) 567 self._jobs.append(job) 568 self.n_dispatched_batches += 1 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __init__(self, batch) 178 # Don't delay the application, to avoid keeping the input 179 # arguments in memory --> 180 self.results = batch() 181 182 def get(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in **call**(self) 70 71 def **call**(self): ---> 72 return [func(_args, *_kwargs) for func, args, kwargs in self.items] 73 74 def **len**(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in (.0) 70 71 def **call**(self): ---> 72 return [func(_args, *_kwargs) for func, args, kwargs in self.items] 73 74 def **len**(self): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in _estimate_param_scan_worker(estimator, params, X, evaluate, evaluate_args, failfast) 142 # deal with result 143 if evaluate is None: # we want full models --> 144 res.append(estimator.model) 145 # we want to evaluate function(s) of the model 146 elif _types.is_iterable(evaluate): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self) 373 except AttributeError: 374 raise AttributeError( --> 375 'Model has not yet been estimated. Call estimate(X) or fit(X) first') AttributeError: Model has not yet been estimated. Call estimate(X) or fit(X) first | — You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
P.S.: quick workaround: use multiple calls to estimate_hidden_markov_model()
Am 08/04/16 um 18:09 schrieb Guillermo Pérez-Hernández:
pyemma.version '2.1-9-g4936d0d' This happened:
its[pp]= pyemma.msm.timescales_hmsm([dtraj[::rs]for dtrajin cl[pp].dtrajs],4, lags=lags_fr[rs].astype(int),nits=5)
|AttributeError Traceback (most recent call last) /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self) 371 try: --> 372 return self._model 373 except AttributeError: AttributeError: 'MaximumLikelihoodHMSM' object has no attribute '_model' During handling of the above exception, another exception occurred: AttributeError Traceback (most recent call last)
in () 3 for pp in prot: 4 pp = pp[:2] ----> 5 its[pp] = pyemma.msm.timescales_hmsm([dtraj[::rs] for dtraj in cl[pp].dtrajs], 4, lags=lags_fr[rs].astype(int), nits=5) 6 7 /home/mi/gph82/programs/PyEmma/pyemma/msm/api.py in timescales_hmsm(dtrajs, nstates, lags, nits, reversible, stationary, connectivity, mincount_connectivity, separate, errors, nsamples, stride, n_jobs, show_progress) 848 itsobj = _ImpliedTimescales(estimator, lags=lags, nits=nits, n_jobs=n_jobs, 849 show_progress=show_progress) --> 850 itsobj.estimate(dtrajs) 851 return itsobj 852 /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate(self, X, **params) 341 if params: 342 self.set_params(**params) --> 343 self._model = self._estimate(X) 344 self._estimated = True 345 return self /home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/implied_timescales.py in _estimate(self, data) 146 self._models, self._estimators = estimate_param_scan(self.estimator, data, param_sets, failfast=False, 147 return_estimators=True, n_jobs=self.n_jobs, --> 148 progress_reporter=self) 149 150 ### PROCESS RESULTS /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate_param_scan(estimator, X, param_sets, evaluate, evaluate_args, failfast, return_estimators, n_jobs, progress_reporter) 296 297 # container for model or function evaluations --> 298 res = pool(task_iter) 299 300 if progress_reporter is not None: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __call__(self, iterable) 798 # was dispatched. In particular this covers the edge 799 # case of Parallel used with an exhausted iterator. --> 800 while self.dispatch_one_batch(iterator): 801 self._iterating = True 802 else: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in dispatch_one_batch(self, iterator) 656 return False 657 else: --> 658 self._dispatch(tasks) 659 return True 660 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in _dispatch(self, batch) 564 565 if self._pool is None: --> 566 job = ImmediateComputeBatch(batch) 567 self._jobs.append(job) 568 self.n_dispatched_batches += 1 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __init__(self, batch) 178 # Don't delay the application, to avoid keeping the input 179 # arguments in memory --> 180 self.results = batch() 181 182 def get(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in **call**(self) 70 71 def **call**(self): ---> 72 return [func(_args, *_kwargs) for func, args, kwargs in self.items] 73 74 def **len**(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in (.0) 70 71 def **call**(self): ---> 72 return [func(_args, *_kwargs) for func, args, kwargs in self.items] 73 74 def **len**(self): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in _estimate_param_scan_worker(estimator, params, X, evaluate, evaluate_args, failfast) 142 # deal with result 143 if evaluate is None: # we want full models --> 144 res.append(estimator.model) 145 # we want to evaluate function(s) of the model 146 elif _types.is_iterable(evaluate): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self) 373 except AttributeError: 374 raise AttributeError( --> 375 'Model has not yet been estimated. Call estimate(X) or fit(X) first') AttributeError: Model has not yet been estimated. Call estimate(X) or fit(X) first | — You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
Hmm, I don't understand what's going on here. On the latest devel, the following commands work fine. Please try them in your install:
>>> from pyemma import msm
>>> import numpy as np
>>> dtraj = [0,1,1,0,0,0,1,1,0,0,0,1,2,2,2,2,2,2,2,2,2,1,1,0,0,0,1,1,0,1,0]
>>> ts = msm.timescales_hmsm(dtraj, 2, [1,2,3,4])
estimating MaximumLikelihoodHMSM: 100% (4/4)
[#############################################] eta 00:00
>>> ts.timescales
array([[ 5.78572184],
[ 5.1429104 ],
[ 4.43976251],
[ 3.67739079]])
Am 08/04/16 um 18:09 schrieb Guillermo Pérez-Hernández:
pyemma.version '2.1-9-g4936d0d' This happened:
its[pp]= pyemma.msm.timescales_hmsm([dtraj[::rs]for dtrajin cl[pp].dtrajs],4, lags=lags_fr[rs].astype(int),nits=5)
|AttributeError Traceback (most recent call last) /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self) 371 try: --> 372 return self._model 373 except AttributeError: AttributeError: 'MaximumLikelihoodHMSM' object has no attribute '_model' During handling of the above exception, another exception occurred: AttributeError Traceback (most recent call last)
in () 3 for pp in prot: 4 pp = pp[:2] ----> 5 its[pp] = pyemma.msm.timescales_hmsm([dtraj[::rs] for dtraj in cl[pp].dtrajs], 4, lags=lags_fr[rs].astype(int), nits=5) 6 7 /home/mi/gph82/programs/PyEmma/pyemma/msm/api.py in timescales_hmsm(dtrajs, nstates, lags, nits, reversible, stationary, connectivity, mincount_connectivity, separate, errors, nsamples, stride, n_jobs, show_progress) 848 itsobj = _ImpliedTimescales(estimator, lags=lags, nits=nits, n_jobs=n_jobs, 849 show_progress=show_progress) --> 850 itsobj.estimate(dtrajs) 851 return itsobj 852 /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate(self, X, **params) 341 if params: 342 self.set_params(**params) --> 343 self._model = self._estimate(X) 344 self._estimated = True 345 return self /home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/implied_timescales.py in _estimate(self, data) 146 self._models, self._estimators = estimate_param_scan(self.estimator, data, param_sets, failfast=False, 147 return_estimators=True, n_jobs=self.n_jobs, --> 148 progress_reporter=self) 149 150 ### PROCESS RESULTS /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate_param_scan(estimator, X, param_sets, evaluate, evaluate_args, failfast, return_estimators, n_jobs, progress_reporter) 296 297 # container for model or function evaluations --> 298 res = pool(task_iter) 299 300 if progress_reporter is not None: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __call__(self, iterable) 798 # was dispatched. In particular this covers the edge 799 # case of Parallel used with an exhausted iterator. --> 800 while self.dispatch_one_batch(iterator): 801 self._iterating = True 802 else: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in dispatch_one_batch(self, iterator) 656 return False 657 else: --> 658 self._dispatch(tasks) 659 return True 660 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in _dispatch(self, batch) 564 565 if self._pool is None: --> 566 job = ImmediateComputeBatch(batch) 567 self._jobs.append(job) 568 self.n_dispatched_batches += 1 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __init__(self, batch) 178 # Don't delay the application, to avoid keeping the input 179 # arguments in memory --> 180 self.results = batch() 181 182 def get(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in **call**(self) 70 71 def **call**(self): ---> 72 return [func(_args, *_kwargs) for func, args, kwargs in self.items] 73 74 def **len**(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in (.0) 70 71 def **call**(self): ---> 72 return [func(_args, *_kwargs) for func, args, kwargs in self.items] 73 74 def **len**(self): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in _estimate_param_scan_worker(estimator, params, X, evaluate, evaluate_args, failfast) 142 # deal with result 143 if evaluate is None: # we want full models --> 144 res.append(estimator.model) 145 # we want to evaluate function(s) of the model 146 elif _types.is_iterable(evaluate): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self) 373 except AttributeError: 374 raise AttributeError( --> 375 'Model has not yet been estimated. Call estimate(X) or fit(X) first') AttributeError: Model has not yet been estimated. Call estimate(X) or fit(X) first | — You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
will do and report back, thks
Wanted to mention that we have the same issue in another part. Noelia and me both have had it the last days.
As to the question of version, I also believe I only started seeing it on 2.1.x. I was worried that it might again be a problem with statistics at extreme lag times. Just head over to our issue if you want trajectories. Noelia seems to have some apparently, although I don't know if they are confidential.
Thanks! I think this is not a lagtime issue. Guille has also seen that but I couldn't reproduce it so far. If you have a not too large example raising this (maybe only some trajectories), it would be great if you could post it.
Am 20/04/16 um 20:45 schrieb Stefan:
As the the question of version, I also believe I only started seeing it on 2.1.x. I was worried that it might again be a problem with statistics at extreme lag times. Just head over to our issue if you want trajectories. Noelia seems to have some apparently, although I don't know if they are publicaly available.
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212554681
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
I have not been able to reproduce with the latest devel, but I will try again tomorrow with a difficult dataset
On 04/20/2016 10:00 PM, Frank Noe wrote:
Thanks! I think this is not a lagtime issue. Guille has also seen that but I couldn't reproduce it so far. If you have a not too large example raising this (maybe only some trajectories), it would be great if you could post it.
Am 20/04/16 um 20:45 schrieb Stefan:
As the the question of version, I also believe I only started seeing it on 2.1.x. I was worried that it might again be a problem with statistics at extreme lag times. Just head over to our issue if you want trajectories. Noelia seems to have some apparently, although I don't know if they are publicaly available.
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212554681
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
Mail: Arnimallee 6, 14195 Berlin, Germany
— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212580513
Dr. Guillermo Pérez-Hernández Freie Universität Berlin Institute for Mathematics Arnimallee 6 D-14195 Berlin tel 0049 30 838 75775
yeap, this is still happening, and the dataset is not fancy or anything...
print(lags)
print([len(dtraj) for dtraj in dtrajs])
pyemma.msm.timescales_hmsm(dtrajs,
nstates=4,
lags=lags)
[4, 8, 12, 16]
[261, 137, 145, 321, 125, 65, 373, 153, 177, 373, 117, 153, 409, 169, 109, 345, 177, 133, 113, 197]
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self)
371 try:
--> 372 return self._model
373 except AttributeError:
AttributeError: 'MaximumLikelihoodHMSM' object has no attribute '_model'
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
<ipython-input-61-8ead127d5f77> in <module>()
3 pyemma.msm.timescales_hmsm(dtrajs,
4 nstates=4,
----> 5 lags=lags)
/home/mi/gph82/programs/PyEmma/pyemma/msm/api.py in timescales_hmsm(dtrajs, nstates, lags, nits, reversible, stationary, connectivity, mincount_connectivity, separate, errors, nsamples, stride, n_jobs, show_progress)
848 itsobj = _ImpliedTimescales(estimator, lags=lags, nits=nits, n_jobs=n_jobs,
849 show_progress=show_progress)
--> 850 itsobj.estimate(dtrajs)
851 return itsobj
852
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate(self, X, **params)
341 if params:
342 self.set_params(**params)
--> 343 self._model = self._estimate(X)
344 self._estimated = True
345 return self
/home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/implied_timescales.py in _estimate(self, data)
146 self._models, self._estimators = estimate_param_scan(self.estimator, data, param_sets, failfast=False,
147 return_estimators=True, n_jobs=self.n_jobs,
--> 148 progress_reporter=self)
149
150 ### PROCESS RESULTS
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate_param_scan(estimator, X, param_sets, evaluate, evaluate_args, failfast, return_estimators, n_jobs, progress_reporter)
296
297 # container for model or function evaluations
--> 298 res = pool(task_iter)
299
300 if progress_reporter is not None:
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __call__(self, iterable)
798 # was dispatched. In particular this covers the edge
799 # case of Parallel used with an exhausted iterator.
--> 800 while self.dispatch_one_batch(iterator):
801 self._iterating = True
802 else:
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in dispatch_one_batch(self, iterator)
656 return False
657 else:
--> 658 self._dispatch(tasks)
659 return True
660
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in _dispatch(self, batch)
564
565 if self._pool is None:
--> 566 job = ImmediateComputeBatch(batch)
567 self._jobs.append(job)
568 self.n_dispatched_batches += 1
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __init__(self, batch)
178 # Don't delay the application, to avoid keeping the input
179 # arguments in memory
--> 180 self.results = batch()
181
182 def get(self):
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __call__(self)
70
71 def __call__(self):
---> 72 return [func(*args, **kwargs) for func, args, kwargs in self.items]
73
74 def __len__(self):
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in <listcomp>(.0)
70
71 def __call__(self):
---> 72 return [func(*args, **kwargs) for func, args, kwargs in self.items]
73
74 def __len__(self):
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in _estimate_param_scan_worker(estimator, params, X, evaluate, evaluate_args, failfast)
142 # deal with result
143 if evaluate is None: # we want full models
--> 144 res.append(estimator.model)
145 # we want to evaluate function(s) of the model
146 elif _types.is_iterable(evaluate):
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self)
373 except AttributeError:
374 raise AttributeError(
--> 375 'Model has not yet been estimated. Call estimate(X) or fit(X) first')
AttributeError: Model has not yet been estimated. Call estimate(X) or fit(X) first
​```
thanks, will look into it.
Am 21/04/16 um 11:00 schrieb Guillermo Pérez-Hernández:
yeap, this is still happening, and the dataset is not fancy or anything...
print(lags) print([len(dtraj)for dtrajin dtrajs]) pyemma.msm.timescales_hmsm(dtrajs, nstates=4, lags=lags)
AttributeError Traceback (most recent call last) /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin model(self) 371 try: --> 372 return self._model 373 except AttributeError:
AttributeError:'MaximumLikelihoodHMSM' object has no attribute'_model'
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
in () 3 pyemma.msm.timescales_hmsm(dtrajs, 4 nstates=4, ----> 5 lags=lags) /home/mi/gph82/programs/PyEmma/pyemma/msm/api.pyin timescales_hmsm(dtrajs, nstates, lags, nits, reversible, stationary, connectivity, mincount_connectivity, separate, errors, nsamples, stride, n_jobs, show_progress) 848 itsobj= _ImpliedTimescales(estimator,lags=lags,nits=nits,n_jobs=n_jobs, 849 show_progress=show_progress) --> 850 itsobj.estimate(dtrajs) 851 return itsobj 852 /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin estimate(self,X,**params) 341 if params: 342 self.set_params(**params) --> 343 self._model= self._estimate(X) 344 self._estimated= True 345 return self /home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/implied_timescales.pyin _estimate(self, data) 146 self._models,self._estimators= estimate_param_scan(self.estimator, data, param_sets,failfast=False, 147 return_estimators=True,n_jobs=self.n_jobs, --> 148 progress_reporter=self) 149 150 ### PROCESS RESULTS /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin estimate_param_scan(estimator,X, param_sets, evaluate, evaluate_args, failfast, return_estimators, n_jobs, progress_reporter) 296 297 # container for model or function evaluations --> 298 res= pool(task_iter) 299 300 if progress_reporteris not None: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin **call**(self, iterable) 798 # was dispatched. In particular this covers the edge 799 # case of Parallel used with an exhausted iterator. --> 800 while self.dispatch_one_batch(iterator): 801 self._iterating= True 802 else: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin dispatch_one_batch(self, iterator) 656 return False 657 else: --> 658 self._dispatch(tasks) 659 return True 660 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin _dispatch(self, batch) 564 565 if self._poolis None: --> 566 job= ImmediateComputeBatch(batch) 567 self._jobs.append(job) 568 self.n_dispatched_batches+= 1 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin **init**(self, batch) 178 # Don't delay the application, to avoid keeping the input 179 # arguments in memory --> 180 self.results= batch() 181 182 def get(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin **call**(self) 70 71 def **call**(self): ---> 72 return [func(_args,_*kwargs)for func, args, kwargsin self.items] 73 74 def **len**(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin (.0) 70 71 def **call**(self): ---> 72 return [func(_args,_*kwargs)for func, args, kwargsin self.items] 73 74 def **len**(self): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin _estimate_param_scan_worker(estimator, params,X, evaluate, evaluate_args, failfast) 142 # deal with result 143 if evaluateis None:# we want full models --> 144 res.append(estimator.model) 145 # we want to evaluate function(s) of the model 146 elif _types.is_iterable(evaluate): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin model(self) 373 except AttributeError: 374 raise AttributeError( --> 375 'Model has not yet been estimated. Call estimate(X) or fit(X) first') AttributeError: Model hasnot yet been estimated. Call estimate(X)or fit(X) first ​``` — You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212820705
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
do you want the sample data?
yes
Am 21/04/16 um 11:05 schrieb Guillermo Pérez-Hernández:
do you want the sample data?
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212821934
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
/group/ag_cmb/simulation-data/dtrajs.hmm.npy
Ran into the same error using MSM.cktest:
iMSM
MaximumLikelihoodMSM(connectivity='largest', count_mode='sliding',
dt_traj='1 step', lag=120, maxerr=1e-08, maxiter=1000000,
reversible=True, sparse=False, statdist_constraint=None)
iMSM.cktest(4)
pyemma.msm.estimators.lagged_model_validators.ChapmanKolmogorovValidator[0] WARNING Changed mlags as some mlags exceeded maximum trajectory length.
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self)
371 try:
--> 372 return self._model
373 except AttributeError:
AttributeError: 'MaximumLikelihoodMSM' object has no attribute '_model'
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
<ipython-input-66-24a54f53dbe0> in <module>()
----> 1 iMSM.cktest(4, )
/home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/maximum_likelihood_msm.py in cktest(self, nsets, memberships, mlags, conf, err_est, show_progress)
798 memberships = self.metastable_memberships
799 ck = ChapmanKolmogorovValidator(self, self, memberships, mlags=mlags, conf=conf, err_est=err_est)
--> 800 ck.estimate(self._dtrajs_full)
801 return ck
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate(self, X, **params)
341 if params:
342 self.set_params(**params)
--> 343 self._model = self._estimate(X)
344 self._estimated = True
345 return self
/home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/lagged_model_validators.py in _estimate(self, data)
133 estimated_models, estimators = \
134 estimate_param_scan(self.test_estimator, data, pargrid, return_estimators=True, failfast=False,
--> 135 progress_reporter=progress_reporter, n_jobs=self.n_jobs)
136 if include0:
137 estimated_models = [None] + estimated_models
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in estimate_param_scan(estimator, X, param_sets, evaluate, evaluate_args, failfast, return_estimators, n_jobs, progress_reporter)
296
297 # container for model or function evaluations
--> 298 res = pool(task_iter)
299
300 if progress_reporter is not None:
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __call__(self, iterable)
798 # was dispatched. In particular this covers the edge
799 # case of Parallel used with an exhausted iterator.
--> 800 while self.dispatch_one_batch(iterator):
801 self._iterating = True
802 else:
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in dispatch_one_batch(self, iterator)
656 return False
657 else:
--> 658 self._dispatch(tasks)
659 return True
660
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in _dispatch(self, batch)
564
565 if self._pool is None:
--> 566 job = ImmediateComputeBatch(batch)
567 self._jobs.append(job)
568 self.n_dispatched_batches += 1
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __init__(self, batch)
178 # Don't delay the application, to avoid keeping the input
179 # arguments in memory
--> 180 self.results = batch()
181
182 def get(self):
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in __call__(self)
70
71 def __call__(self):
---> 72 return [func(*args, **kwargs) for func, args, kwargs in self.items]
73
74 def __len__(self):
/home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.py in <listcomp>(.0)
70
71 def __call__(self):
---> 72 return [func(*args, **kwargs) for func, args, kwargs in self.items]
73
74 def __len__(self):
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in _estimate_param_scan_worker(estimator, params, X, evaluate, evaluate_args, failfast)
142 # deal with result
143 if evaluate is None: # we want full models
--> 144 res.append(estimator.model)
145 # we want to evaluate function(s) of the model
146 elif _types.is_iterable(evaluate):
/home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.py in model(self)
373 except AttributeError:
374 raise AttributeError(
--> 375 'Model has not yet been estimated. Call estimate(X) or fit(X) first')
AttributeError: Model has not yet been estimated. Call estimate(X) or fit(X) first
I think I know what's the problem. I can fix this soon.
Am 21/04/16 um 11:24 schrieb Guillermo Pérez-Hernández:
Ran into the same error using MSM.cktest:
iMSM MaximumLikelihoodMSM(connectivity='largest',count_mode='sliding', dt_traj='1 step',lag=120,maxerr=1e-08,maxiter=1000000, reversible=True,sparse=False,statdist_constraint=None) iMSM.cktest(4)
pyemma.msm.estimators.lagged_model_validators.ChapmanKolmogorovValidator[0]WARNING Changed mlagsas some mlags exceeded maximum trajectory length.
AttributeError Traceback (most recent call last) /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin model(self) 371 try: --> 372 return self._model 373 except AttributeError:
AttributeError:'MaximumLikelihoodMSM' object has no attribute'_model'
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
in () ----> 1 iMSM.cktest(4, ) /home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/maximum_likelihood_msm.pyin cktest(self, nsets, memberships, mlags, conf, err_est, show_progress) 798 memberships= self.metastable_memberships 799 ck= ChapmanKolmogorovValidator(self,self, memberships,mlags=mlags,conf=conf,err_est=err_est) --> 800 ck.estimate(self._dtrajs_full) 801 return ck /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin estimate(self,X,**params) 341 if params: 342 self.set_params(**params) --> 343 self._model= self._estimate(X) 344 self._estimated= True 345 return self /home/mi/gph82/programs/PyEmma/pyemma/msm/estimators/lagged_model_validators.pyin _estimate(self, data) 133 estimated_models, estimators= \ 134 estimate_param_scan(self.test_estimator, data, pargrid,return_estimators=True,failfast=False, --> 135 progress_reporter=progress_reporter,n_jobs=self.n_jobs) 136 if include0: 137 estimated_models= [None]+ estimated_models /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin estimate_param_scan(estimator,X, param_sets, evaluate, evaluate_args, failfast, return_estimators, n_jobs, progress_reporter) 296 297 # container for model or function evaluations --> 298 res= pool(task_iter) 299 300 if progress_reporteris not None: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin **call**(self, iterable) 798 # was dispatched. In particular this covers the edge 799 # case of Parallel used with an exhausted iterator. --> 800 while self.dispatch_one_batch(iterator): 801 self._iterating= True 802 else: /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin dispatch_one_batch(self, iterator) 656 return False 657 else: --> 658 self._dispatch(tasks) 659 return True 660 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin _dispatch(self, batch) 564 565 if self._poolis None: --> 566 job= ImmediateComputeBatch(batch) 567 self._jobs.append(job) 568 self.n_dispatched_batches+= 1 /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin **init**(self, batch) 178 # Don't delay the application, to avoid keeping the input 179 # arguments in memory --> 180 self.results= batch() 181 182 def get(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin **call**(self) 70 71 def **call**(self): ---> 72 return [func(_args,_*kwargs)for func, args, kwargsin self.items] 73 74 def **len**(self): /home/mi/gph82/miniconda3/lib/python3.4/site-packages/joblib-0.9.4-py3.4.egg/joblib/parallel.pyin (.0) 70 71 def **call**(self): ---> 72 return [func(_args,_*kwargs)for func, args, kwargsin self.items] 73 74 def **len**(self): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin _estimate_param_scan_worker(estimator, params,X, evaluate, evaluate_args, failfast) 142 # deal with result 143 if evaluateis None:# we want full models --> 144 res.append(estimator.model) 145 # we want to evaluate function(s) of the model 146 elif _types.is_iterable(evaluate): /home/mi/gph82/programs/PyEmma/pyemma/_base/estimator.pyin model(self) 373 except AttributeError: 374 raise AttributeError( --> 375 'Model has not yet been estimated. Call estimate(X) or fit(X) first') AttributeError: Model hasnot yet been estimated. Call estimate(X)or fit(X) first — You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212828119
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
Stefan, did you install 2.1 with conda? Can you check which bhmm version you have?
trying now with bhmm.version '0.6.1'
will report back
yeap, this is now gone....
I forced the update of bhmm via conda install pyemma immediately followed by conda remove pyemma (to go back to the devel version).
@marscher and @noe: What would be the "proper" way of doing this? Any way one can trigger this update automatically when doing python setup.py develop on pyemma?
Frank, the issue is not in bhmm. In our case we don't use bhmms. We use MaximumLikelihoodMSM
. Check the pastebin if you want.
@stefdoerr , can you check if this persists with the workaround proposed above?
@gph82 actually we have enforced the upgrade to 0.6, via the constraint "bhmm >=0.6,<0.7", which is both present in conda recipe and setup.py.
hmm that's weird, because I was running 0.5, @marscher , I pull+python setup.py develop almost daily
OK. Leave this issue open. Guille's issue is due to the fact that his bhmm wasn't up to date. This would have not occurred if he had installed PyEMMA via conda, but I think thre is still a dependency problem via pip install, because that should have triggered a bhmm upgrade when installing the PyEMMA devel version.
The issue reported by Stefan is indirectly triggered when at least a single estimate in the timescales convenience function fails. In Stefan's and Guille's example they are triggered for different reasons.
Am 21/04/16 um 16:50 schrieb Martin K. Scherer:
@gph82 https://github.com/gph82 actually we have enforced the upgrade to 0.6, via the constraint "bhmm >=0.6,<0.7", which is both present in conda recipe and setup.py.
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212955755
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
Putting the bhmm dependency problem into a separate issue...
Am 21/04/16 um 16:54 schrieb Guillermo Pérez-Hernández:
hmm that's weird, because I was running 0.5, @marscher https://github.com/marscher , I pull+python setup.py develop almost daily
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212957839
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
Why does a single failure of estimate silently fail everything? What was the argument of failfast?
That's not supposed to happen. It's a bug.
Am 21/04/16 um 17:02 schrieb Martin K. Scherer:
Why does a single failure of estimate silently fail everything? What was the argument of failfast?
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212961192
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
:+1: Default value for failfast is true, so it should raise. Maybe the estimator itself does not raise, but just print a warning, which is might be never printed?
With the updated bhmm (0.6.1), the problem does not appear when calling hmm ( https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212820705) but persists when calling cktest(https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-212828119)
Here is a minimal example producing that problem:
from pyemma import msm
hmm = msm.estimate_hidden_markov_model([0, 0, 0, 1, 1, 1, 0, 0], 2, 1)
hmm.cktest()
I think the problem has been introduced by changing the behavior of the Estimator base class. It could be fixed by handling the exception in ImpliedTimescales, ChapmanKolgmorovTest etc, or by putting the Estimator back to the previous behavior. Not sure yet what's the best solution here.
Am 25.04.2016 um 13:37 schrieb Frank Noe:
Here is a minimal example producing that problem:
from pyemmaimport msm hmm= msm.estimate_hidden_markov_model([0,0,0,1,1,1,0,0],2,1) hmm.cktest()
I think the problem has been introduced by changing the behavior of the Estimator base class. It could be fixed by handling the exception in ImpliedTimescales, ChapmanKolgmorovTest etc, or by putting the Estimator back to the previous behavior. Not sure yet what's the best solution here.
— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-214276681
with 2.0.4 it raises an AssertionError without any message. I guess that this change in behaviour has been introduced via the switch from joblib 0.8.4 (which caused other bugs, found by Tim Hempel) to 0.9.4.
25-04-16 13:43:05 pyemma.msm.estimators.lagged_model_validators.ChapmanKolmogorovValidator[0] WARNING Changed mlags as some mlags exceeded maximum trajectory length.
estimating MaximumLikelihoodHMSM: 25% (2/8) [############################ ] eta 00:01 -Traceback (most recent call last):
File "test_cktest.py", line 3, in <module>
hmm.cktest()
File "/home/marscher/workspace/pyemma/pyemma/msm/estimators/maximum_likelihood_hmsm.py", line 317, in cktest
ck.estimate(self._dtrajs_full)
File "/home/marscher/workspace/pyemma/pyemma/_base/estimator.py", line 341, in estimate
self._model = self._estimate(X)
File "/home/marscher/workspace/pyemma/pyemma/msm/estimators/lagged_model_validators.py", line 137, in _estimate
n_jobs=self.n_jobs)
File "/home/marscher/workspace/pyemma/pyemma/_base/estimator.py", line 299, in estimate_param_scan
res = pool(task_iter)
File "/home/marscher/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 653, in __call__
self.dispatch(function, args, kwargs)
File "/home/marscher/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 400, in dispatch
job = ImmediateApply(func, args, kwargs)
File "/home/marscher/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 138, in __init__
self.results = func(*args, **kwargs)
File "/home/marscher/workspace/pyemma/pyemma/_base/estimator.py", line 136, in _estimate_param_scan_worker
raise e
AssertionError
I think the problem is this code snippet:
# run estimation
model = None
try: # catch any exception
model = estimator.estimate(X, **params)
except:
e = sys.exc_info()[0]
if failfast:
raise e
else:
pass # just return model=None
The exception handler block re-raises the type of the exception, which is not correct, since the "type" class does not derive from Exception base class.
But in joblib only the case of a proper exception (derived from BaseException) is handled.
OK, what do you suggest?
Am 25/04/16 um 13:59 schrieb Martin K. Scherer:
I think the problem is this code snippet:
# run estimation model = None try: # catch any exception model = estimator.estimate(X, **params) except: e = sys.exc_info()[0] if failfast: raise e else: pass # just return model=None
The exception handler block re-raises the type of the exception, which is not correct, since the "type" class does not derive from Exception base class.
But in joblib only the case of a proper exception (derived from BaseException) is handled.
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/764#issuecomment-214282413
Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin
Phone: (+49) (0)30 838 75354 Web: research.franknoe.de
Please add a test-case with the minimal example. This code:
from pyemma import msm
hmm = msm.estimate_hidden_markov_model([0, 0, 0, 1, 1, 1, 0, 0], 2, 1)
hmm.cktest()
should not raise by default, but should raise when failfast=True.
ok.
I think this issue is solved by #784
Yes.
pyemma.version '2.1-9-g4936d0d' This happened: