General Mixture Models

[arose13]

Are general mixture models coming anytime soon? I am currently using pomegranate for this, however, I would love to be able to get credible intervals for the parameters found.

[ferrine]

Does this class fit your needs?

[arose13]

Thanks for the reply. I'm trying to now give me a moment.

[arose13]

with pm.Model() as iq_model:
    # Priors
    p_a = pm.Uniform('a weight', lower=0, upper=1)
    p_b = pm.Deterministic('b weight', 1 - p_a)

    a = pm.Normal('a', mu=100, sd=10)
    b = pm.Normal('b', mu=150, sd=10)

    # Model
    total = pm.Mixture('iq', w=[p_a, p_b], comp_dists=[a, b], observed=iq)

    # Sample
    trace = pm.sample(10000)

    pm.traceplot(trace)
    graph.show()

This gives me this error that's rather opaque to me.

AsTensorError: ('Cannot convert <bound method _tensor_py_operators.mean of a> to TensorType', <class 'method'>)

[ferrine]

could you please post the full traceback?

[junpenglao]

I think a better starting point is the following notebooks: https://github.com/pymc-devs/pymc3/blob/master/docs/source/notebooks/gaussian_mixture_model.ipynb https://github.com/pymc-devs/pymc3/blob/master/docs/source/notebooks/gaussian-mixture-model-advi.ipynb and https://github.com/pymc-devs/pymc3/blob/master/docs/source/notebooks/dp_mix.ipynb

But I think currently these are not optimal for problems in higher dimensions, I tried it with a 2D DP-GMM but did not get a good fitting at the time (a few months ago).

[arose13]

@junpenglao I've seen and tried those already and they don't allow for the inclusion of non-normal distributions.

@ferrine

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/home/anthony/miniconda3/lib/python3.6/site-packages/pymc3/distributions/mixture.py in _comp_means(self)
     87         try:
---> 88             return tt.as_tensor_variable(self.comp_dists.mean)
     89         except AttributeError:

AttributeError: 'list' object has no attribute 'mean'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/type.py in dtype_specs(self)
    269                 'complex64': (complex, 'theano_complex64', 'NPY_COMPLEX64')
--> 270             }[self.dtype]
    271         except KeyError:

KeyError: 'object'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/basic.py in constant_or_value(x, rtype, name, ndim, dtype)
    249             rval = rtype(
--> 250                 TensorType(dtype=x_.dtype, broadcastable=bcastable),
    251                 x_.copy(),

/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/type.py in __init__(self, dtype, broadcastable, name, sparse_grad)
     50         self.broadcastable = tuple(bool(b) for b in broadcastable)
---> 51         self.dtype_specs()  # error checking is done there
     52         self.name = name

/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/type.py in dtype_specs(self)
    272             raise TypeError("Unsupported dtype for %s: %s"
--> 273                             % (self.__class__.__name__, self.dtype))
    274 

TypeError: Unsupported dtype for TensorType: object

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/basic.py in as_tensor_variable(x, name, ndim)
    205     try:
--> 206         return constant(x, name=name, ndim=ndim)
    207     except TypeError:

/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/basic.py in constant(x, name, ndim, dtype)
    263     ret = constant_or_value(x, rtype=TensorConstant, name=name, ndim=ndim,
--> 264                             dtype=dtype)
    265 

/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/basic.py in constant_or_value(x, rtype, name, ndim, dtype)
    258     except Exception:
--> 259         raise TypeError("Could not convert %s to TensorType" % x, type(x))
    260 

TypeError: ('Could not convert <bound method _tensor_py_operators.mean of a> to TensorType', <class 'method'>)

During handling of the above exception, another exception occurred:

AsTensorError                             Traceback (most recent call last)
<ipython-input-19-1dad83a4a623> in <module>()
      8 
      9     # Model
---> 10     total = pm.Mixture('iq', w=[p_a, p_b], comp_dists=[a, b])#, observed=iq)
     11 #     total = pm.NormalMixture(
     12 #         'iq',

/home/anthony/miniconda3/lib/python3.6/site-packages/pymc3/distributions/distribution.py in __new__(cls, name, *args, **kwargs)
     28         if isinstance(name, string_types):
     29             data = kwargs.pop('observed', None)
---> 30             dist = cls.dist(*args, **kwargs)
     31             return model.Var(name, dist, data)
     32         else:

/home/anthony/miniconda3/lib/python3.6/site-packages/pymc3/distributions/distribution.py in dist(cls, *args, **kwargs)
     39     def dist(cls, *args, **kwargs):
     40         dist = object.__new__(cls)
---> 41         dist.__init__(*args, **kwargs)
     42         return dist
     43 

/home/anthony/miniconda3/lib/python3.6/site-packages/pymc3/distributions/mixture.py in __init__(self, w, comp_dists, *args, **kwargs)
     53 
     54             try:
---> 55                 self.mean = (w * self._comp_means()).sum(axis=-1)
     56 
     57                 if 'mean' not in defaults:

/home/anthony/miniconda3/lib/python3.6/site-packages/pymc3/distributions/mixture.py in _comp_means(self)
     89         except AttributeError:
     90             return tt.stack([comp_dist.mean for comp_dist in self.comp_dists],
---> 91                             axis=1)
     92 
     93     def _comp_modes(self):

/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/basic.py in stack(*tensors, **kwargs)
   4583         dtype = scal.upcast(*[i.dtype for i in tensors])
   4584         return theano.tensor.opt.MakeVector(dtype)(*tensors)
-> 4585     return join(axis, *[shape_padaxis(t, axis) for t in tensors])
   4586 
   4587 

/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/basic.py in <listcomp>(.0)
   4583         dtype = scal.upcast(*[i.dtype for i in tensors])
   4584         return theano.tensor.opt.MakeVector(dtype)(*tensors)
-> 4585     return join(axis, *[shape_padaxis(t, axis) for t in tensors])
   4586 
   4587 

/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/basic.py in shape_padaxis(t, axis)
   4470 
   4471     """
-> 4472     _t = as_tensor_variable(t)
   4473 
   4474     ndim = _t.ndim + 1

/home/anthony/miniconda3/lib/python3.6/site-packages/theano/tensor/basic.py in as_tensor_variable(x, name, ndim)
    210         except Exception:
    211             str_x = repr(x)
--> 212         raise AsTensorError("Cannot convert %s to TensorType" % str_x, type(x))
    213 
    214 # this has a different name, because _as_tensor_variable is the

AsTensorError: ('Cannot convert <bound method _tensor_py_operators.mean of a> to TensorType', <class 'method'>)

[junpenglao]

could you please also post the data?

[ferrine]

I suppose comp_dists should be smth like [Normal.dist(...), ...]

[AustinRochford]

@ferrine is correct, need to pass the distribution, not an RV to comp_dists.

[arose13]

@ferrine @AustinRochford

Okay it doesn't crash and I get it to sample but only I don't give it observed data. Does that mean I cannot run inference on mixtures?

[twiecki]

@arose13 no that should be supported.

[arose13]

My bad I was not specific enough. Is it possible to get the trace of the parameters of the underlying distributions in a Mixture since you are not allowed to give Mixture RVs?

[AustinRochford]

@arose13 That is quite possible; you can still pass in RVs as parameters to the component distributions. See this gist for an example.

[arose13]

Nice! An example this simple should be in the docs 😄 .

Thank you very much for the help!

[AustinRochford]

Documentation pull requests welcome ;)

[arose13]

I don't mind doing that