Bayesian Regularization

[Ohayoosan]

Can you add Bayesian Regularization to neupy?

[itdxer]

Do you have references to some papers that apply this technique in the context of neural networks?

[Ohayoosan]

Here some of reference to bayesian regularization. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0188553

[itdxer]

To be honest, I'm a bit skeptical. It looks like BR is quite slow (because of the inverse hessian). It adds improvement, but I'm not sure whether the same improvement won't be observed with simple L2 regularisation. Also, I don't see any prove and set of the assumptions, that has been made for BR (they just have brief overview of the algorithm). Also, in the modified cost function, they use squared error (could be because LM algorithm requires it) but I'm not sure whether it will work for any other function.

Do you know some paper that compares BR to some other regularisation methods like L2 or L1 regularisation? (didn't find anything useful after quick search)

[Ohayoosan]

Unfortunately, I did not find the journal you mentioned. I am new to the neural network and I got the assignment from my lecturer to make BR neural network using python as main language. I struggled on how to adding the modified cost function to Levenberg-Marquardt function in neupy. Sorry for bothered you.

[itdxer]

It's a bit tricky for the LM algorithm, since at each training step algorithm expects that loss function is MSE. It's possible to update loss function, even thought it's a bit hacky

import tensorflow as tf
from neupy import algorithms
from neupy.layers import *

def modified_mse(actual, expected):
    # Loss funtion can be modified here
    return tf.reduce_mean(tf.square(actual - expected))

class LMwithBR(algorithms.LevenbergMarquardt):
    loss = None

    def __init__(self, *args, **kwargs):
        self.loss = modified_mse
        super(LMwithBR, self).__init__(*args, **kwargs)

optimizer = LMwithBR(Input(10) >> Sigmoid(5))

[itdxer]

I'm closing this issue for now, since I don't plan adding BR in the library any time soon. Feel free to open new ticket if you have some problems with the code that I've added in my previous comment.

[wangq-ntu]

It's a bit tricky for the LM algorithm, since at each training step algorithm expects that loss function is MSE. It's possible to update loss function, even thought it's a bit hacky

import tensorflow as tf
from neupy import algorithms
from neupy.layers import *

def modified_mse(actual, expected):
    # Loss funtion can be modified here
    return tf.reduce_mean(tf.square(actual - expected))

class LMwithBR(algorithms.LevenbergMarquardt):
    loss = None

    def __init__(self, *args, **kwargs):
        self.loss = modified_mse
        super(LMwithBR, self).__init__(*args, **kwargs)

optimizer = LMwithBR(Input(10) >> Sigmoid(5))

I am also try to using Lm with BR in pytorch. is your code can use in pytorch?

[itdxer]

unfortunately NeuPy works only with tensorflow and the same code won't work in pytorch

[wangq-ntu]

unfortunately NeuPy works only with tensorflow and the same code won't work in pytorch

Ok, Thanks.