search

dattalab / pyhsmm-library-models

library models built on top of pyhsmm
0 stars 1 forks source link

Cannot train models on very large datasets #30

Closed alexbw closed 11 years ago

alexbw commented 11 years ago

Seems to be an error within pickle.

As a possible solution, joblib.dump() supports splitting up arrays into multiple files.

from sklearn.externals import joblib to get at it, if you want to see.

Traceback (most recent call last):
  File "/home/abw11/Code/pymouse/scripts/build_model.py", line 252, in <module>
    baker.run()
  File "/home/abw11/anaconda/lib/python2.7/site-packages/baker.py", line 823, in run
    value = self.apply(*self.parse(argv), instance=instance)
  File "/home/abw11/anaconda/lib/python2.7/site-packages/baker.py", line 801, in apply
    return cmd.fn(*newargs, **newkwargs)
  File "/home/abw11/Code/pymouse/scripts/build_model.py", line 155, in build
    lhsmmModel.fit(X=data, X_held_out=data_test)
  File "/home/abw11/Code/pymouse/lhsmm.py", line 372, in fit
    self.hsmm_model.add_data(X, left_censoring=self.left_censoring)
  File "/home/abw11/Code/pyhsmm_library_models/pyhsmm/models.py", line 446, in add_data
    **kwargs))
  File "/home/abw11/Code/pyhsmm_library_models/library_models.py", line 161, in __init__
    precomputed_likelihoods = kwargs['model'].obs_distns[0].get_all_likelihoods(data)
  File "/home/abw11/Code/pyhsmm_library_models/library_models.py", line 45, in get_all_likelihoods
    cPickle.dump((likelihoods,shifted_likelihoods,maxes),outfile,protocol=-1)
SystemError: error return without exception set
Traceback (most recent call last):
  File "/home/abw11/Code/pymouse/scripts/build_model.py", line 252, in <module>
    baker.run()
  File "/home/abw11/anaconda/lib/python2.7/site-packages/baker.py", line 823, in run
    value = self.apply(*self.parse(argv), instance=instance)
  File "/home/abw11/anaconda/lib/python2.7/site-packages/baker.py", line 801, in apply
    return cmd.fn(*newargs, **newkwargs)
  File "/home/abw11/Code/pymouse/scripts/build_model.py", line 155, in build
    lhsmmModel.fit(X=data, X_held_out=data_test)
  File "/home/abw11/Code/pymouse/lhsmm.py", line 372, in fit
    self.hsmm_model.add_data(X, left_censoring=self.left_censoring)
  File "/home/abw11/Code/pyhsmm_library_models/pyhsmm/models.py", line 446, in add_data
    **kwargs))
  File "/home/abw11/Code/pyhsmm_library_models/library_models.py", line 161, in __init__
    precomputed_likelihoods = kwargs['model'].obs_distns[0].get_all_likelihoods(data)
  File "/home/abw11/Code/pyhsmm_library_models/library_models.py", line 45, in get_all_likelihoods
    cPickle.dump((likelihoods,shifted_likelihoods,maxes),outfile,protocol=-1)
SystemError: error return without exception set
Traceback (most recent call last):
  File "/home/abw11/Code/pymouse/scripts/build_model.py", line 252, in <module>
    baker.run()
  File "/home/abw11/anaconda/lib/python2.7/site-packages/baker.py", line 823, in run
    value = self.apply(*self.parse(argv), instance=instance)
  File "/home/abw11/anaconda/lib/python2.7/site-packages/baker.py", line 801, in apply
    return cmd.fn(*newargs, **newkwargs)
  File "/home/abw11/Code/pymouse/scripts/build_model.py", line 155, in build
    lhsmmModel.fit(X=data, X_held_out=data_test)
  File "/home/abw11/Code/pymouse/lhsmm.py", line 372, in fit
    self.hsmm_model.add_data(X, left_censoring=self.left_censoring)
  File "/home/abw11/Code/pyhsmm_library_models/pyhsmm/models.py", line 446, in add_data
    **kwargs))
  File "/home/abw11/Code/pyhsmm_library_models/library_models.py", line 161, in __init__
    precomputed_likelihoods = kwargs['model'].obs_distns[0].get_all_likelihoods(data)
  File "/home/abw11/Code/pyhsmm_library_models/library_models.py", line 45, in get_all_likelihoods
    cPickle.dump((likelihoods,shifted_likelihoods,maxes),outfile,protocol=-1)
SystemError: error return without exception set