search

pymc-devs / pymc

Bayesian Modeling and Probabilistic Programming in Python
https://docs.pymc.io/
Other
8.63k stars 1.99k forks source link

Vectors of multivariate variables #535

Closed fonnesbeck closed 5 years ago

fonnesbeck commented 10 years ago

It would be useful if we could model multiple independent multivariate variables in the same statement. For example, if I wanted four multivariate normal vectors with the same prior, I should be able to specify:

f = pm.MvNormal('f', np.zeros(3), np.eye(3), shape=(4,3))

but it currently returns a ValueError complaining of non-aligned matrices.

fonnesbeck commented 8 years ago

That is one solution, I suppose. I do like the idea of having a PyMC object representing the vector of variables, rather than a list, but perhaps I'm just being picky. In other words, treating a multivariate distribution the same way we treat a univariate distribution.

fonnesbeck commented 8 years ago

I'm trying to use the list comprehension approach above, for example:

        p_uae = [pm.Dirichlet('p_uae_%i' % i, θ, shape=n_outcomes) for i,θ in enumerate(θ_uae)]

however, I get errors suggesting that this is not cool with Theano:

/Users/fonnescj/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py in __setitem__(self, item, value)
    501                     raise TypeError("Ambiguous name: %s - please check the "
    502                                     "names of the inputs of your function "
--> 503                                     "for duplicates." % str(item))
    504                 if isinstance(s, gof.Container):
    505                     s.value = value

TypeError: Ambiguous name: p_uae_30_stickbreaking - please check the names of the inputs of your function for duplicates.

Prior to the error, I am also flooded with Theano warnings that look like this:

/Users/fonnescj/anaconda3/lib/python3.5/site-packages/theano/gradient.py:545: UserWarning: grad method was asked to compute the gradient with respect to a variable that is not part of the computational graph of the cost, or is used only by a non-differentiable operator: p_6_stickbreaking
  handle_disconnected(elem)

As far as I can tell, there should be nothing in the model that is ambiguously named. Here is the full model:

import theano.tensor as T

SumTo1 = pm.transforms.SumTo1()
inverse_logit = pm.transforms.inverse_logit

def specify_model(model):

    with model:

        # Impute followup times
        followup_time = pm.Uniform('followup_time', followup_min, followup_max, 
                                   shape=len(followup_min), 
                                   observed=followup_masked)

        # Mean probabilities (on logit scale)
        μ = pm.Normal('μ', 0, 0.01, shape=n_outcomes)
        # Followup time covariates 
        β_fup = pm.Normal('β_fup', 0, 0.01, shape=n_outcomes)
        # Age covariate
        β_age = pm.Normal('β_age', 0, 0.01, shape=n_outcomes)

        # Study random effect
        τ = pm.Exponential('τ', 0.1, testval=1)
        ϵ = pm.Normal('ϵ', 0, τ, shape=n_studies)

        # Expected value (on logit scale)
        θ_uae = [T.exp(μ + β_fup*followup_time[i] + β_age*age_centered[i] + ϵ[study_index[i]]) 
                         for i in range(arms)]

        # Inverse-logit transformation to convert to probabilities
        p_uae = [pm.Dirichlet('p_uae_%i' % i, t, shape=n_outcomes) for i,t in enumerate(θ_uae)]

        # Multinomial data likelihood
        likelihood = [pm.Multinomial('likelihood_%i' % i, followup_n[i], p_uae[i], 
                                     observed=outcomes[i]) for i in range(arms)]

        p_6 = pm.Dirichlet('p_6', T.exp(μ + β_fup*6), shape=n_outcomes)
        p_12 = pm.Dirichlet('p_12', T.exp(μ + β_fup*12), shape=n_outcomes)
        p_6_50 = pm.Dirichlet('p_6_50', T.exp(μ + β_fup*6 + β_age*10), shape=n_outcomes)

    return model

There is no naming ambiguity that I can see.

twiecki commented 8 years ago

Hm, yeah, that looks pretty good to me. Have you tried T.concatenate to convert them to vectors? I'd also be curious if @nouiz thinks that a list-comprehension of theano expressions isn't supposed to work.

nouiz commented 8 years ago

I didn't found what line was tried for list-comprehension that didn't worked. Can you give me that line?

List-comprehension work when Theano is able to infer the shape of the variable. But we didn't implemented all cases. Also, in the dev version of Theano, there is more case implemented then in the last release 0.7. This work for example:

import theano
x=theano.tensor.matrix()
[i+j for i,j in enumerate(x.shape)]
fonnesbeck commented 8 years ago

It happens in the p_uae list comprehension. Says there is an ambiguous name despite being uniquely named via enumerate indices.

fonnesbeck commented 8 years ago

Using concatenate or stacklists does not seem to help either:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-92-882bbf1c3e98> in <module>()
----> 1 uae_model = specify_model(uae_model, 'uae')
      2 
      3 with uae_model:
      4 
      5     if True:

<ipython-input-91-98403b254ade> in specify_model(model, intervention)
     46         # Inverse-logit transformation to convert to probabilities
     47         π = T.concatenate([pm.Dirichlet('π_%i' % i, t, shape=n_outcomes) 
---> 48                           for i,t in enumerate(θ_uae)])
     49 
     50         # Multinomial data likelihood

<ipython-input-91-98403b254ade> in <listcomp>(.0)
     45 
     46         # Inverse-logit transformation to convert to probabilities
---> 47         π = T.concatenate([pm.Dirichlet('π_%i' % i, t, shape=n_outcomes) 
     48                           for i,t in enumerate(θ_uae)])
     49 

/Users/fonnescj/anaconda3/lib/python3.5/site-packages/theano/tensor/var.py in __iter__(self)
    546     def __iter__(self):
    547         try:
--> 548             for i in xrange(theano.tensor.basic.get_vector_length(self)):
    549                 yield self[i]
    550         except TypeError:

/Users/fonnescj/anaconda3/lib/python3.5/site-packages/theano/tensor/basic.py in get_vector_length(v)
   4282     else:
   4283         msg = str(v)
-> 4284     raise ValueError("length not known: %s" % msg)
   4285 
   4286
fonnesbeck commented 8 years ago

Even without concatenating, and also dropping back to Metropolis sampling, it does not go well.

Miles-Garnsey commented 8 years ago

How do these proposals play with constructs like the randomwalk structures which have a natural shape?

I might be running across some of the issues described here in a new issue I raised #960.

nouiz commented 8 years ago

You can not always loop on a Theano variable. Doing it via a for loop or list comprehensions is the same.

Theano need to infer the shape when the loop is executed and we don't do it for many cases. If you know the shape, you can do it manually, but looping for the number of iteration you need and do the indexing.

replace: [... for i, v in enumerate(o_uae)]

by

[... for i in range(N)]

and replace v by o_uae[i] in the ...

junpenglao commented 8 years ago

My personal experience: I applied the code similar @fonnesbeck and it works really well under Metropolis

kobs = [pm.Multinomial('kobs_%i' % i, p = theta[:,i], n=Nitem, 
                                     observed=k[i]) for i in np.arange(Nsubj)]

However, using NUTS it returns the following Theano error, which I am not able to track down.

ERROR (theano.gof.opt): SeqOptimizer apply MergeOptimizer
ERROR:theano.gof.opt:SeqOptimizer apply MergeOptimizer
ERROR (theano.gof.opt): Traceback:
ERROR:theano.gof.opt:Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
    clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
    clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR (theano.gof.opt): SeqOptimizer apply MergeOptimizer
ERROR:theano.gof.opt:SeqOptimizer apply MergeOptimizer
ERROR (theano.gof.opt): Traceback:
ERROR:theano.gof.opt:Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
    clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
    clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

But the sampling result is correct as well. If you guys find it helpful I can post the full model I used.

nouiz commented 8 years ago

Can you update Theano to the dev version?

http://www.deeplearning.net/software/theano/install.html#bleeding-edge-install-instructions

If that don't fix it, I need a way to reproduce it. You can use the function_dump method in Theano:

http://deeplearning.net/software/theano/library/compile/function.html#theano.compile.function.function_dump

On Fri, Feb 26, 2016 at 8:49 AM, Junpeng Lao notifications@github.com wrote:

My personal experience: I applied the code similar @fonnesbeck https://github.com/fonnesbeck and it works really well under Metropolis

kobs = [pm.Multinomial('kobs_%i' % i, p = theta[:,i], n=Nitem, observed=k[i]) for i in np.arange(Nsubj)]

However, using NUTS it returns the following Theano error, which I am not able to track down.

ERROR (theano.gof.opt): SeqOptimizer apply MergeOptimizer ERROR:theano.gof.opt:SeqOptimizer apply MergeOptimizer ERROR (theano.gof.opt): Traceback: ERROR:theano.gof.opt:Traceback: ERROR (theano.gof.opt): Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize ret = self.apply(fgraph, _args, *_kwargs) File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply clients = pairs[0][0].clients + pairs[0][1].clients AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize ret = self.apply(fgraph, _args, *_kwargs) File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply clients = pairs[0][0].clients + pairs[0][1].clients AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR (theano.gof.opt): SeqOptimizer apply MergeOptimizer ERROR:theano.gof.opt:SeqOptimizer apply MergeOptimizer ERROR (theano.gof.opt): Traceback: ERROR:theano.gof.opt:Traceback: ERROR (theano.gof.opt): Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize ret = self.apply(fgraph, _args, *_kwargs) File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply clients = pairs[0][0].clients + pairs[0][1].clients AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize ret = self.apply(fgraph, _args, *_kwargs) File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply clients = pairs[0][0].clients + pairs[0][1].clients AttributeError: 'TensorVariable' object has no attribute 'clients'

But the sampling result is correct as well. If you guys find it helpful I can post the full model I used.

— Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-189284686.

junpenglao commented 8 years ago

Thanks @nouiz, I am already using the dev branch. I will try to use the function_dump method.

psinger commented 8 years ago

I am wondering whether there is already a solution for this? How can I simply specify multiple Dirichlet distributions at once (e.g., for a mixture)?

PietJones commented 8 years ago

I am also trying to get see how to implement a dirichlet process that can be applied to a matrix/dataframe, trying to extend the existing example of the Dirichlet Process. 

N = df.shape[0] #df has a shape of 400,1000
K = 30

with pm.Model() as model:    
    alpha = pm.Gamma('alpha', 1., 1.)
    beta = pm.Beta('beta', 1., alpha, shape=K)
    w = pm.Deterministic('w', beta * T.concatenate([[1], T.extra_ops.cumprod(1 - beta)[:-1]]))
    component = pm.Categorical('component', w, shape=N)

    alpha_w = pm.Gamma('alpha_w',1.,1.,shape=K)
    target_beta  = pm.Beta('targets', alpha_w,1., shape=K)
    likelihood = pm.Beta('like', alpha_w[component], 1.,observed=df.values)
    #likelihood = pm.Beta('like', T.concatenate([alpha_w[component] for ii in range(df.shape[1])]), 1.,observed=df.values)
    #likelihood = [pm.Beta('m_%i' %ii, alpha_w[component[ii]],1., observed=df.values[ii,:]) for ii in range(N)]

    step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,likelihood])
    step2 = pm.ElemwiseCategoricalStep([component], np.arange(K))

    trace_ = pm.sample(20000, [step1, step2])

I have tried the three definitions of the likelihood, but they all fail. The first one gives an inconsistent shape error. The second produces a similar error to what Chris got in his gist. The third gives me a theano error: Attribute list has not attribute owner, when trying to assign the step method.

Sorry if I this is not the correct topic, but the information in this thread seems to be related. Any help would be much appreciated.

nouiz commented 8 years ago

Can you give me the full Theano error for the 3rd case?

On Wed, May 4, 2016 at 10:55 PM, PietJones notifications@github.com wrote:

I am also trying to get see how to implement a dirichlet process that can be applied to a matrix/dataframe, trying to extend the existing example of the Dirichlet Process.

N = df.shape[0] #df has a shape of 400,1000 K = 30

with pm.Model() as model: alpha = pm.Gamma('alpha', 1., 1.) beta = pm.Beta('beta', 1., alpha, shape=K) w = pm.Deterministic('w', beta * T.concatenate([[1], T.extra_ops.cumprod(1 - beta)[:-1]])) component = pm.Categorical('component', w, shape=N)

alpha_w = pm.Gamma('alpha_w',1.,1.,shape=K)
target_beta  = pm.Beta('targets', alpha_w,1., shape=K)
likelihood = pm.Beta('like', alpha_w[component], 1.,observed=df.values)
#likelihood = pm.Beta('like', T.concatenate([alpha_w[component] for ii in range(df.shape[1])]), 1.,observed=df.values)
#likelihood = [pm.Beta('m_%i' %ii, alpha_w[component[ii]],1., observed=df.values[ii,:]) for ii in range(N)]

step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,likelihood])
step2 = pm.ElemwiseCategoricalStep([component], np.arange(K))

trace_ = pm.sample(20000, [step1, step2])

I have tried the three definitions of the likelihood, but they all fail. The first one gives an inconsistent shape error. The second produces a similar error to what Chris got in his gist. The third gives me a theano error: Attribute list has not attribute owner, when trying to assign the step method.

Sorry if I this is not the correct topic, but the information in this thread seems to be related. Any help would be much appreciated.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217064072

PietJones commented 8 years ago

Hi here it is, sorry about the formatting:

AttributeError Traceback (most recent call last)

in () 15 obs = [pm.Beta('m_%i' %ii, alpha_w[component[ii]],1., observed=df.values[ii,:]) for ii in range(N)] 16 ---> 17 step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,obs]) 18 step2 = pm.ElemwiseCategoricalStep([component], np.arange(K)) 19 /Users/jq2/miniconda/envs/pymc3/lib/python2.7/site-packages/pymc3-3.0-py2.7.egg/pymc3/step_methods/arraystep.pyc in **new**(cls, _args, *_kwargs) 46 47 #get the actual inputs from the vars ---> 48 vars = inputvars(vars) 49 50 if not blocked and len(vars) > 1: /Users/jq2/miniconda/envs/pymc3/lib/python2.7/site-packages/pymc3-3.0-py2.7.egg/pymc3/theanof.pyc in inputvars(a) 19 r : list of tensor variables that are inputs 20 """ ---> 21 return [v for v in inputs(makeiter(a)) if isinstance(v, t.TensorVariable)] 22 23 def cont_inputs(f): /Users/jq2/git/Theano/theano/gof/graph.py in inputs(variable_list, blockers) 695 696 """ --> 697 vlist = ancestors(variable_list, blockers) 698 rval = [r for r in vlist if r.owner is None] 699 return rval /Users/jq2/git/Theano/theano/gof/graph.py in ancestors(variable_list, blockers) 674 if r.owner and (not blockers or r not in blockers): 675 return reversed(r.owner.inputs) --> 676 dfs_variables = stack_search(deque(variable_list), expand, 'dfs') 677 return dfs_variables 678 /Users/jq2/git/Theano/theano/gof/graph.py in stack_search(start, expand, mode, build_inv) 642 rval_list.append(l) 643 rval_set.add(id(l)) --> 644 expand_l = expand(l) 645 if expand_l: 646 if build_inv: /Users/jq2/git/Theano/theano/gof/graph.py in expand(r) 672 """ 673 def expand(r): --> 674 if r.owner and (not blockers or r not in blockers): 675 return reversed(r.owner.inputs) 676 dfs_variables = stack_search(deque(variable_list), expand, 'dfs') AttributeError: 'list' object has no attribute 'owner'
nouiz commented 8 years ago

The first Theano function called is:

/Users/jq2/git/Theano/theano/gof/graph.py in inputs(variable_list, blockers)

I think the problem is that variable_list or blockers aren't a list of Theano variable. One of the element is probably a list instead of a Theano variable.

Can you verify that with pdb? If so, the fix will be in pymc3

On Thu, May 5, 2016 at 9:34 AM, PietJones notifications@github.com wrote:

AttributeError Traceback (most recent call last) in () 15 obs = [pm.Beta('m_%i' %ii, alpha_w[component[ii]],1., observed=df.values[ii,:]) for ii in range(N)] 16 ---> 17 step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,obs]) 18 step2 = pm.ElemwiseCategoricalStep([component], np.arange(K)) 19

/Users/jq2/miniconda/envs/pymc3/lib/python2.7/site-packages/pymc3-3.0-py2.7.egg/pymc3/step_methods/arraystep.pyc in new(cls, _args, *_kwargs) 46 47 #get the actual inputs from the vars ---> 48 vars = inputvars(vars) 49 50 if not blocked and len(vars) > 1:

/Users/jq2/miniconda/envs/pymc3/lib/python2.7/site-packages/pymc3-3.0-py2.7.egg/pymc3/theanof.pyc in inputvars(a) 19 r : list of tensor variables that are inputs 20 """ ---> 21 return [v for v in inputs(makeiter(a)) if isinstance(v, t.TensorVariable)] 22 23 def cont_inputs(f):

/Users/jq2/git/Theano/theano/gof/graph.py in inputs(variable_list, blockers) 695 696 """ --> 697 vlist = ancestors(variable_list, blockers) 698 rval = [r for r in vlist if r.owner is None] 699 return rval

/Users/jq2/git/Theano/theano/gof/graph.py in ancestors(variable_list, blockers) 674 if r.owner and (not blockers or r not in blockers): 675 return reversed(r.owner.inputs) --> 676 dfs_variables = stack_search(deque(variable_list), expand, 'dfs') 677 return dfs_variables 678

/Users/jq2/git/Theano/theano/gof/graph.py in stack_search(start, expand, mode, build_inv) 642 rval_list.append(l) 643 rval_set.add(id(l)) --> 644 expand_l = expand(l) 645 if expand_l: 646 if build_inv:

/Users/jq2/git/Theano/theano/gof/graph.py in expand(r) 672 """ 673 def expand(r): --> 674 if r.owner and (not blockers or r not in blockers): 675 return reversed(r.owner.inputs) 676 dfs_variables = stack_search(deque(variable_list), expand, 'dfs')

AttributeError: 'list' object has no attribute 'owner'

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217155671

PietJones commented 8 years ago

Tried pdb, not sure I am doing it right, but it gave the following, after inserting:

pdb.set_trace()

Before the step1 assignment

Applied log-transform to alpha and added transformed alpha_log to model.
Applied logodds-transform to beta and added transformed beta_logodds to model.
Applied log-transform to alpha_w and added transformed alpha_w_log to model.
Applied logodds-transform to targets and added transformed targets_logodds to model.
> /Users/jq2/Downloads/TestPYMC3.py(45)<module>()
 -> step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,likelihood])
(Pdb) n
 AttributeError: "'list' object has no attribute 'owner'"
> /Users/jq2/Downloads/TestPYMC3.py(45)<module>()
-> step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,likelihood])

(Pdb) type(likelihood)
<type 'list'>
(Pdb) type(target_beta)
<class 'pymc3.model.TransformedRV'>
(Pdb) type(likelihood[0])
<class 'pymc3.model.ObservedRV'>
twiecki commented 8 years ago

@PietJones You shouldn't include observed variables to be sampled.

nouiz commented 8 years ago

Can PyMC3 give a better user error for that case?

On Thu, May 5, 2016 at 10:21 AM, Thomas Wiecki notifications@github.com wrote:

@PietJones https://github.com/PietJones You shouldn't include observed variables to be sampled.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217167357

PietJones commented 8 years ago

Cool, I took that from:

http://austinrochford.com/posts/2016-02-25-density-estimation-dpm.html

After changing, now I get the following error:

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpYXDK_O/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

Is there some size limit that I am not aware of? The data frame is not that large: (450, 1051)

nouiz commented 8 years ago

You can use this PR for a work around:

https://github.com/Theano/Theano/pull/4289

On Thu, May 5, 2016 at 10:30 AM, PietJones notifications@github.com wrote:

Cool, I took that from:

http://austinrochford.com/posts/2016-02-25-density-estimation-dpm.html http://url

After changing, now I get the following error:

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpYXDK_O/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

Is there some size limit that I am not aware of? The data frame is not that large: (450, 1051)

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217169460

PietJones commented 8 years ago

@nouiz Thnx for the advice, again not sure if this was what you meant that I should do, but I tried the following, and I still get the same error:

cd ~/git
git clone https://github.com/Theano/Theano
git fetch origin pull/4289/head:pr-4289
git checkout pr-4289
python setup.py develop

I then restarted my ipython/jupyter kernel and reran my code.

nouiz commented 8 years ago

Delete your Theano cache. If that don't fix it, you probably using the old Theano. Uninstall Theano many times to be sure it is not installed and reinstall as you just did.

On Thu, May 5, 2016 at 11:05 AM, PietJones notifications@github.com wrote:

@nouiz https://github.com/nouiz Thnx for the advice, again not sure if this was what you meant that I should do, but I tried the following, and I still get the same error:

cd ~/git git clone https://github.com/Theano/Theano git fetch origin pull/4289/head:pr-4289 git checkout pr-4289 python setup.py develop

I then restarted my ipython/jupyter kernel and reran my code.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217178862

PietJones commented 8 years ago

I tried the following.

rm -r ~/.theano*
pip uninstall theano #did this several times until there was error
cd ~/git/theano #then fetched the PR, did git checkout etc
python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.__version__"
0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using jupyter (did restart the kernel), (don't have cuda). Sorry for the trouble.

nouiz commented 8 years ago

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send me the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones notifications@github.com wrote:

I tried the following.

rm -r ~/.theano* pip uninstall theano #did this several times until there was error cd ~/git/theano #then fetched the PR, did git checkout etc python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version" 0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using jupyter (did restart the kernel), (don't have cuda). Sorry for the trouble.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605

PietJones commented 8 years ago

Hi,

Find attached the mod.cpp file which failed to compile.

https://gist.github.com/PietJones/8e53946b2738008095ced8fb9ab4db44

I dont think the entire file uploaded, just in case here is google drive link:

https://drive.google.com/file/d/0B2e7WGnBljbJZnJ1T1NDU1FjS1k/view?usp=sharing

On Thu, May 5, 2016 at 1:00 PM, Frédéric Bastien notifications@github.com wrote:

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send me the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones notifications@github.com wrote:

I tried the following.

rm -r ~/.theano* pip uninstall theano #did this several times until there was error cd ~/git/theano #then fetched the PR, did git checkout etc python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version" 0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using jupyter (did restart the kernel), (don't have cuda). Sorry for the trouble.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217210834

nouiz commented 8 years ago

Update Theano to 0.8.2. I have the impression that you use an older version.

Fred

On Thu, May 5, 2016 at 1:25 PM, PietJones notifications@github.com wrote:

Hi,

Find attached the mod.cpp file which failed to compile.

On Thu, May 5, 2016 at 1:00 PM, Frédéric Bastien <notifications@github.com

wrote:

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send me the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones notifications@github.com wrote:

I tried the following.

rm -r ~/.theano* pip uninstall theano #did this several times until there was error cd ~/git/theano #then fetched the PR, did git checkout etc python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version" 0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1):

/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using jupyter (did restart the kernel), (don't have cuda). Sorry for the trouble.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217210834

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217216992

PietJones commented 8 years ago

I originally had that version of Theano, which gave the same error. The older version I posted about above was using a specific Pull Request to see if that would help.

P

On Fri, May 6, 2016 at 9:03 AM, Frédéric Bastien notifications@github.com wrote:

Update Theano to 0.8.2. I have the impression that you use an older version.

Fred

On Thu, May 5, 2016 at 1:25 PM, PietJones notifications@github.com wrote:

Hi,

Find attached the mod.cpp file which failed to compile.

On Thu, May 5, 2016 at 1:00 PM, Frédéric Bastien < notifications@github.com

wrote:

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send me the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones notifications@github.com wrote:

I tried the following.

rm -r ~/.theano* pip uninstall theano #did this several times until there was error cd ~/git/theano #then fetched the PR, did git checkout etc python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version" 0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1):

/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32:

fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using jupyter (did restart the kernel), (don't have cuda). Sorry for the trouble.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub < https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605>

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217210834

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217216992

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217433674

nouiz commented 8 years ago

Can you confirm it was the pull request about the GpuJoin proble on windows ( https://github.com/Theano/Theano/pull/4289)? https://github.com/Theano/Theano/pull/4289

I check that code and it would have tested what I wanted to test.

Can you manually apply this diff and test again?

""" diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py index cd74c1e..e9b44b5 100644 --- a/theano/tensor/opt.py +++ b/theano/tensor/opt.py @@ -6761,7 +6761,7 @@ def elemwise_max_input_fct(node):

inputs.

 if not theano.config.cxx:
     return 31
  • return 1024

  • return 512

    local_elemwise_fusion = local_elemwise_fusion_op(T.Elemwise, """

If it still fail, instead of a max of 512, try 256, 128, ...

Tell me the biggest number that work.

On Fri, May 6, 2016 at 9:47 AM, PietJones notifications@github.com wrote:

I originally had that version of Theano, which gave the same error. The older version I posted about above was using a specific Pull Request to see if that would help.

P

On Fri, May 6, 2016 at 9:03 AM, Frédéric Bastien <notifications@github.com

wrote:

Update Theano to 0.8.2. I have the impression that you use an older version.

Fred

On Thu, May 5, 2016 at 1:25 PM, PietJones notifications@github.com wrote:

Hi,

Find attached the mod.cpp file which failed to compile.

On Thu, May 5, 2016 at 1:00 PM, Frédéric Bastien < notifications@github.com

wrote:

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send me the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones <notifications@github.com

wrote:

I tried the following.

rm -r ~/.theano* pip uninstall theano #did this several times until there was error cd ~/git/theano #then fetched the PR, did git checkout etc python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version" 0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1):

/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32:

fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using jupyter (did restart the kernel), (don't have cuda). Sorry for the trouble.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub < https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605>

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub < https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217210834>

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217216992

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217433674

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217444869

PietJones commented 8 years ago

Thnx for the advice, I tried all of the above, editing the file manually, removing the .theano directory, then restarting the jupyter kernel and running the code again, still get the same error. This is also further down before the actual traceback:

/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpEMrDR6  /mod.cpp:35943:32: fatal error: bracket nesting level exceeded maximum of 256
    if (!PyErr_Occurred()) {
                           ^
/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpEMrDR6/mod.cpp:35943:32: note: use -fbracket-depth=N to increase maximum nesting level
1 error generated.
nouiz commented 8 years ago

Which new value did you try? Only 512? Can you try something like 31? If it still fait with 31, then try this diff:

diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py
index a08e900..ef0821d 100644
--- a/theano/tensor/opt.py
+++ b/theano/tensor/opt.py
@@ -6724,6 +6724,8 @@ def local_add_mul_fusion(node):
                 isinstance(inp.owner.op.scalar_op, s_op)):
             l = list(node.inputs)
             l.remove(inp)
+            if len(l) + len(inp.owner.inputs) > 31:
+                return
             output_node = node.op(*(l + inp.owner.inputs))

             copy_stack_trace(node.outputs[0], output_node)

This opt could also cause this extra big Elemwise.

PietJones commented 8 years ago

I have tried 1024, 512, 256 and 31, they all result in the same problem. Not sure what correction you want me to implement, as the formatting of what you sent has been corrupted. But the changes that I tried was :

5549 def local_add_mul_fusion(node): 5550 """Fuse consecutive add or mul in one such node with more inputs. 5551 5552 It is better to fuse add/mul that way then in a Composite node as 5553 this make the inner graph of the Compiste smaller. This allow to 5554 put more computation in a Composite before hitting the max 5555 recusion limit when pickling Composite. 5556 5557 """ 5558 if (not isinstance(node.op, Elemwise) or 5559 not isinstance(node.op.scalar_op, (scalar.Add, scalar.Mul))): 5560 return False 5561 5562 s_op = node.op.scalar_op.class 5563 for inp in node.inputs: 5564 if (inp.owner and 5565 isinstance(inp.owner.op, Elemwise) and 5566 isinstance(inp.owner.op.scalar_op, sop)): 5567 l = list(node.inputs) 5568 l.remove(inp) 5569 if len(l) + len(inp.owner.inputs) > 31: 5570 return 5571 #return [node.op((l + inp.owner.inputs))] 5572 outputnode = node.op((l + inp.owner.inputs)) 5573 copy_stack_trace(node.ouput[0],output_node) 5574

which still gave an error: https://gist.github.com/PietJones/26339593d2e7862ef60881ea09a817cb

On Tue, May 10, 2016 at 10:16 AM, Frédéric Bastien <notifications@github.com

wrote:

Which new value did you try? Only 512? Can you try something like 31? If it still fait with 31, then try this diff:

diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py index a08e900..ef0821d 100644 --- a/theano/tensor/opt.py +++ b/theano/tensor/opt.py @@ -6724,6 +6724,8 @@ def local_add_mul_fusion(node): isinstance(inp.owner.op.scalar_op, s_op)): l = list(node.inputs) l.remove(inp)

  • if len(l) + len(inp.owner.inputs) > 31:

        return
output_node = node.op(*(l + inp.owner.inputs))

copy_stack_trace(node.outputs[0], output_node)

This opt could also cause this extra big Elemwise.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/pymc-devs/pymc3/issues/535#issuecomment-218170906

twiecki commented 7 years ago

@fonnesbeck I think this works for Multivariate now, right?

fonnesbeck commented 7 years ago

A list comprehension seems to work now, yes. Ultimately I'd like to be able to specify a vector of multivariates using the shape argument, as in the original issue, but that will be for post-3.0.

twiecki commented 7 years ago

I think that should also work, no? At least for 3D multivariates.

fonnesbeck commented 7 years ago

It runs, but does not "work":

from pymc3 import *

with Model():

    p = Dirichlet('p', np.ones(3), shape=(4,3))
    x = Multinomial('x', np.array([20, 16, 10, 5]), p, shape=(4,3))
    print('p initial:', p.tag.test_value)
    print('x initial:', x.tag.test_value)
    tr = sample(50)
    print('x final:', x.tag.test_value)

yields:

p initial: [[ 0.33333333  0.33333333  0.33333333]
 [ 0.33333333  0.33333333  0.33333333]
 [ 0.33333333  0.33333333  0.33333333]
 [ 0.33333333  0.33333333  0.33333333]]
x initial: [[7 7 7]
 [5 5 5]
 [3 3 3]
 [2 2 2]]
Assigned NUTS to p_stickbreaking_
Assigned Metropolis to x
100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 287.60it/s]
x final: [[7 7 7]
 [5 5 5]
 [3 3 3]
 [2 2 2]]

So, the x's don't sum to n, yet it does not fail!

fonnesbeck commented 5 years ago

This is tied up in the shape refactoring. Closing.

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.