cossio / RestrictedBoltzmannMachines.jl

Train and sample Restricted Boltzmann machines in Julia
MIT License
15 stars 3 forks source link

pReLU eta must be between -1 and 1 #3

Closed cossio closed 2 years ago

cossio commented 2 years ago

We can enforce this constraint by parameterizing eta, for example:

eta = tanh(xi)

where now xi is unbounded.

We can use something faster than tanh. Here are some examples:

image

This suggests that the faster alternative is:

eta = xi / (1 + abs(xi))
cossio commented 2 years ago

We don't have to parameterize eta. Rather, we can use a different parameter. So, instead of using

image

which is required to be between -1 and 1, we can use:

image

which is unbounded.

The relation between the two is:

image

cossio commented 2 years ago

Closed in https://github.com/cossio/RestrictedBoltzmannMachines.jl/commit/e774670e59889803b483f9da25f734d78d3936c7