Some basic problem within PyMC3.

[Jingfei-Liu]

Description of your problem

Hello everyone! I am a beginneer to PyMC3, and I got a problem when I try to implement Bayesian Neural Networks (BNN) with PyMC3.

The problem is that I want to combine GMM with BNN during the training process of BNN with NUTS. When I run the following simple code, the problem raise.

Please provide a minimal, self-contained, and reproducible example.

import numpy
from sklearn import mixture
import theano
import theano.tensor as T
import pymc3 as pm

input_nodes = 50 #Dimension of the objective problem
hidden_nodes1 = 10
hidden_nodes2 = 10
hidden_nodes3 = 50

loc1=1
scale1=0.2

#Normal Sampling
x_train = scale1*numpy.random.randn(1000,50)+loc1

y_train = 2*x_train+1

def construct_NN(x_train, y_train):

    # set number of nodes in each input,hidden,output layer
    # set number of nodes in each input,hidden,output layer
    inodes  = input_nodes #input layer
    hnodes1 = hidden_nodes1
    hnodes2 = hidden_nodes2
    hnodes3  = hidden_nodes3

    #Initial parameter for NN
    wh1 = numpy.random.normal(0.0, pow(hnodes1,-0.5), (inodes, hnodes1))

    b1  = numpy.random.normal(0.0, pow(hnodes1,-0.5), (hnodes1,))

    wh2 = numpy.random.normal(0.0, pow(hnodes2,-0.5), (hnodes1, hnodes2))

    b2  = numpy.random.normal(0.0, pow(hnodes2,-0.5), (hnodes2,))

    wh3 = numpy.random.normal(0.0, pow(hnodes2,-0.5), (hnodes2, hnodes3))

    b3  = numpy.random.normal(0.0, pow(hnodes2,-0.5), (hnodes3,))

    with pm.Model() as neural_network:

        ann_input = x_train #pm.Data('ann_input1', x_train)
        ann_output = y_train #pm.Data('ann_output1', y_train)

        # Weights from input to hidden layer
        weights_in_1 = pm.Normal('w_in_1', 0, sigma=1,
                                 shape=(input_nodes, hidden_nodes1),
                                 testval=wh1)

        # Bias from input to hidden layer
        weights_in_b = pm.Normal('w_in_b1', 0, sigma=1,
                                 shape=(hidden_nodes1,),
                                 testval=b1)        

        # Weights from 1st to 2nd layer
        weights_in_2 = pm.Normal('w_in_2', 0, sigma=1,
                                shape=(hidden_nodes1, hidden_nodes2),
                                testval=wh2)

        # Bias from input to hidden layer
        weights_in_b2 = pm.Normal('w_in_b2', 0, sigma=1,
                                 shape=(hidden_nodes2,),
                                 testval=b2)     

        # Weights from 2nd to 3rd layer
        weights_in_3 = pm.Normal('w_in_3', 0, sigma=1,
                                shape=(hidden_nodes2, hidden_nodes3),
                                testval=wh3)

        # Bias from 2nd to 3rd layer
        weights_in_b3 = pm.Normal('w_in_b3', 0, sigma=1,
                                 shape=(hidden_nodes3,),
                                 testval=b3)     

        ϵ = pm.HalfCauchy('ϵ', 5)

        ######################################################
        #Fit the Multivariate distribution 
        ######################################################

        hidden_outputs1 = pm.math.tanh(pm.math.dot(ann_input,weights_in_1)+weights_in_b)

        hidden_outputs2 = pm.math.tanh(pm.math.dot(hidden_outputs1,weights_in_2)+weights_in_b2)  

        hidden_outputs3 = pm.math.tanh(pm.math.dot(hidden_outputs2,weights_in_3)+weights_in_b3) 

        #Bayesian Gaussian Mixture Model
        PDF = mixture.BayesianGaussianMixture(n_components=20,n_init=5,
                                              covariance_type='full').fit(hidden_outputs3)

        μ = sum(PDF.means_)

        y_pred = pm.Normal('y_pred', mu=μ, sd=ϵ, observed=ann_output)

    return neural_network

neural_network = construct_NN(x_train, y_train)

with neural_network:

    trace_pce = pm.sample(5000,chains=4)

Please provide the full traceback.

[The error output here]

When I run this code I got the following error information :
"ValueError: setting an array element with a sequence."

I guss this error comes from :
"PDF = mixture.BayesianGaussianMixture(n_components=20,n_init=5,
                                              covariance_type='full').fit(hidden_outputs3)"

Because the at this stage the term "hidden_outputs3" is not an array.  And I don't known how to deal with  this problem. 

Please provide any additional information below.

Could you please give me some advices? Thanks for your help.

Versions and main components

• PyMC3 Version: 3.8
• Theano Version: 1.0.4
• Python Version: 3
• Operating system: Windows 10
• How did you install PyMC3: (conda/pip): pip install

[Jingfei-Liu]

Could you please give some advices? Thanks very much!

[AlexAndorra]

Hi @Jingfei-Liu! As this is more a usage question than a bug report, would you mind asking it on our Discourse please?

[Jingfei-Liu]

Okey, thanks! I will try it. @AlexAndorra