`_stateseq_norep` and others not being set up properly

[mattjj]

Quoth the @alexbw in #47

"I'm on the new dev branch, with sparse matrices that will fit into memory. I'm getting the following error when running the parallel-library-subhmm.py test. library-subhmm.py works, though."

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)<string> in <module>()
/home/abw11/Code/pyhsmm_library_models/pyhsmm/parallel.py in _call(f, data_id, **kwargs)
     86     @engine_global_namespace
     87     def _call(f,data_id,**kwargs):
---> 88         return f(my_data[data_id],**kwargs)
     89
     90     if engine_globals is not None:
/home/abw11/Code/pyhsmm_library_models/library_subhmm_models.py in _state_sampler(frozen_aBl, **kwargs)
    117                 data=frozen_aBl, # dummy
    118                 frozen_aBl=frozen_aBl,
--> 119                 initialize_from_prior=False,temp=temp,**kwargs)
    120         like = global_model.states_list[-1].log_likelihood()
    121         big_stateseq = global_model.states_list.pop().big_stateseq
/home/abw11/Code/pyhsmm_library_models/pyhsmm/models.pyc in add_data(self, data, stateseq, trunc, right_censoring, left_censoring, **kwargs)
    457             left_censoring=left_censoring,
    458             trunc=trunc,
--> 459             **kwargs))
    460
    461     ### generation
/home/abw11/Code/pyhsmm_library_models/library_subhmm_models.pyc in __init__(self, model, data, frozen_aBl, **kwargs)
     37             self._frozen_aBls = [frozen_aBl] * self.hsmm_trans_matrix.shape[0]
     38         super(HSMMIntNegBinVariantFrozenSubHMMsStates,self).__init__(
---> 39                 model=model,data=data,**kwargs)
     40
     41     # TODO compute likelihoods lazily? push this into aBls? why'd I break it
/home/abw11/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in __init__(self, model, *args, **kwargs)
   1119     def __init__(self,model,*args,**kwargs):
   1120         self.model = model
-> 1121         super(HSMMIntNegBinVariantSubHMMsStates,self).__init__(model,*args,**kwargs)
   1122         self.data = self.data.astype('float32',copy=False) if self.data is not None else None
   1123         self._alphan = None
/home/abw11/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in __init__(self, *args, **kwargs)
    735
    736     def __init__(self,*args,**kwargs):
--> 737         HSMMStatesPython.__init__(self,*args,**kwargs)
    738
    739     def clear_caches(self):
/home/abw11/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in __init__(self, model, right_censoring, left_censoring, trunc, stateseq, **kwargs)
    449         self.trunc = trunc
    450
--> 451         super(HSMMStatesPython,self).__init__(model,stateseq=stateseq,**kwargs)
    452
    453     def _get_stateseq(self):
/home/abw11/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in __init__(self, model, T, data, stateseq, initialize_from_prior, **kwargs)
     31         else:
     32             if data is not None and not initialize_from_prior:
---> 33                 self.resample(**kwargs)
     34             else:
     35                 self.generate_states()
/home/abw11/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in resample(self, temp)
   1343     def resample(self,temp=None):
   1344         # TODO something with temperature
-> 1345         self._remove_substates_from_subHMMs()
   1346         alphan = self.messages_forwards_normalized()
   1347         self.sample_backwards_normalized(alphan)
/home/abw11/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in _remove_substates_from_subHMMs(self)
   1355
   1356     def _remove_substates_from_subHMMs(self):
-> 1357         for superstate, states_obj in zip(self.stateseq_norep, self.substates_list):
   1358             self.model.HMMs[superstate].states_list.remove(states_obj)
   1359         self.substates_list = []
/home/abw11/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in stateseq_norep(self)
    463     @property
    464     def stateseq_norep(self):
--> 465         if self._stateseq_norep is None:
    466             self._stateseq_norep, self._durations_censored = rle(self.stateseq)
    467         return self._stateseq_norep
AttributeError: 'HSMMIntNegBinVariantFrozenSubHMMsStates' object has no attribute '_stateseq_norep'

[mattjj]

Ah, it's right in the trace:

-> 1345         self._remove_substates_from_subHMMs()

That's a new function I added, and it's assuming some stuff is already done but it's not. Fixing...

[mattjj]

Tested by running both the parallel and serial correctness tests; things ran and looked reasonable.

[alexbw]

Still broken on Jefferson.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)<string> in <module>()
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/parallel.py in _call(f, data_id, **kwargs)
     86 def has_data(data):
     87     return phash(data) in parallel_state.data_residency
---> 88
     89 def add_data(data,costfunc=len):
     90     parallel_state.check_is_ready()
/home/alexbw/Code/pyhsmm_library_models/library_subhmm_models.py in _state_sampler(frozen_aBl, **kwargs)
    124                 data=frozen_aBl, # dummy
    125                 frozen_aBl=frozen_aBl,
--> 126                 initialize_from_prior=False,temp=temp,**kwargs)
    127         like = global_model.states_list[-1].log_likelihood()
    128         big_stateseq = global_model.states_list.pop().big_stateseq
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/models.pyc in add_data(self, data, stateseq, trunc, right_censoring, left_censoring, **kwargs)
    457             left_censoring=left_censoring,
    458             trunc=trunc,
--> 459             **kwargs))
    460
    461     ### generation
/home/alexbw/Code/pyhsmm_library_models/library_subhmm_models.pyc in __init__(self, model, data, frozen_aBl, **kwargs)
     37             self._frozen_aBls = [frozen_aBl] * self.hsmm_trans_matrix.shape[0]
     38         super(HSMMIntNegBinVariantFrozenSubHMMsStates,self).__init__(
---> 39                 model=model,data=data,**kwargs)
     40
     41     # TODO compute likelihoods lazily? push this into aBls? why'd I break it
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in __init__(self, model, *args, **kwargs)
   1119     def __init__(self,model,*args,**kwargs):
   1120         self.model = model
-> 1121         super(HSMMIntNegBinVariantSubHMMsStates,self).__init__(model,*args,**kwargs)
   1122         self.data = self.data.astype('float32',copy=False) if self.data is not None else None
   1123         self._alphan = None
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in __init__(self, *args, **kwargs)
    735
    736     def __init__(self,*args,**kwargs):
--> 737         HSMMStatesPython.__init__(self,*args,**kwargs)
    738
    739     def clear_caches(self):
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in __init__(self, model, right_censoring, left_censoring, trunc, stateseq, **kwargs)
    449         self.trunc = trunc
    450
--> 451         super(HSMMStatesPython,self).__init__(model,stateseq=stateseq,**kwargs)
    452
    453     def _get_stateseq(self):
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in __init__(self, model, T, data, stateseq, initialize_from_prior, **kwargs)
     31         else:
     32             if data is not None and not initialize_from_prior:
---> 33                 self.resample(**kwargs)
     34             else:
     35                 self.generate_states()
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in resample(self, temp)
   1343     def resample(self,temp=None):
   1344         # TODO something with temperature
-> 1345         self._remove_substates_from_subHMMs()
   1346         alphan = self.messages_forwards_normalized()
   1347         self.sample_backwards_normalized(alphan)
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in _remove_substates_from_subHMMs(self)
   1355
   1356     def _remove_substates_from_subHMMs(self):
-> 1357         for superstate, states_obj in zip(self.stateseq_norep, self.substates_list):
   1358             self.model.HMMs[superstate].states_list.remove(states_obj)
   1359         self.substates_list = []
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in stateseq_norep(self)
    464     def stateseq_norep(self):
    465         if not hasattr(self,"_stateseq_norep") or self._stateseq_norep is None:
--> 466             self._stateseq_norep, self._durations_censored = rle(self.stateseq)
    467         return self._stateseq_norep
    468
/home/alexbw/Code/pyhsmm_library_models/pyhsmm/internals/states.pyc in _get_stateseq(self)
    452
    453     def _get_stateseq(self):
--> 454         return self._stateseq
    455
    456     def _set_stateseq(self,stateseq):
AttributeError: 'HSMMIntNegBinVariantFrozenSubHMMsStates' object has no attribute '_stateseq'

[alexbw]

I'm running real_data_plots/parallel-library-subhmms.py.

[mattjj]

It worked on my machine and my local branches were identical to the remote branches, so this was a code syncing issue between the remote and your local copy.

On Jefferson, in ~alexbw/Code/pyhsmm_library_models/pyhsmm, the command git diff origin subhmms was not empty; there were a lot of differences! So the code wasn't sync'd because your local branch included lots of local commits you had made (and git pull merges things together, instead of replacing).

I moved the subhmms branch with your local commits (not sure what they were, but there were a lot, so we may need things) to a new branch alexbw/subhmms. I then followed this procedure to check out exactly my subhmms branch in your copy of the repository (under the name subhmms).

Then I went to the pyhsmm_library_models directory and ran correctness_tests/library-subhmms-parallel.py and everything worked!

How might we avoid this annoyance in the future? Checking git diffs is a good idea to see if code is really the same. Pulls and merges are complicated when there are local changes.

Maybe we can follow your proposed naming convention of mattjj/dev and alexbw/dev so that mattjj/dev always has my code (always easy to switch to exactly what I'm running) and you can make local changes in alexbw/dev and merge (or replace) into it whenever convenient.

That comes down to making a branch before making local changes, and not committing them to dev (or maybe in the future we'll call it mattjj/dev) unless we're careful about checking diffs.

I also have a feeling that submodules made things complicated... maybe we could get away with using subtrees here (i.e. having pyhsmm as a subtree in pyhsmm-library-models).

REMINDER If you need your local changes that were in pyhsmm:subhmms, they're in pyhsmm:alexbw/subhmms now.

[alexbw]

In summary, I'm a weiner.

On Nov 6, 2013, at 7:41 PM, Matthew Johnson notifications@github.com wrote:

Closed #48.

— Reply to this email directly or view it on GitHub.

[mattjj]

Not at all. I've had the same frustration before and it still took time to figure this out. This issue wasn't something you did; it's something we should figure out how to optimize in our git workflow, which both of us are inventing as we go along!

[mattjj]

http://i.imgur.com/ZKAN9K7.gif

[alexbw]

Gee

On Nov 6, 2013, at 9:48 PM, Matthew Johnson notifications@github.com wrote:

http://i.imgur.com/ZKAN9K7.gif

— Reply to this email directly or view it on GitHub.