Using optimizer with integers

[RichardPortelli]

Hello,

I am trying to find a way to run this very basic example nicely provided by you guys, but with a matrix of integers instead of double: I know I could give a bad Evaluate method resutl if input is not integer but I feel that's not the way to do. I tried to change the type but it triggers that assert : The given MatType must be arma::mat, arma::fmat, arma::sp_mat, or arma::sp_fmat, or it is not known to work! Am I missing something?

include

class SquaredFunction { public: double Evaluate(const arma::mat& x) { return 2 * std::pow(arma::norm(x), 2.0); } };

int main() { arma::mat x("1.0 -1.0 1.0"); ens::SA<> optimizer; SquaredFunction f; // Create function to be optimized. optimizer.Optimize(f, x); std::cout << "Minimum of squared function found with simulated annealing is " << x; }

[zoq]

#define ENS_DISABLE_TYPE_CHECKS
#include <ensmallen.hpp>

class SquaredFunction
{
public:
double Evaluate(const arma::Mat<arma::uword>& x)
{
return 2 * std::pow(arma::norm(x), 2.0);
}
};

int main()
{
arma::Mat<arma::uword> x("1.0 -1.0 1.0");
ens::SA<> optimizer;
SquaredFunction f; // Create function to be optimized.
optimizer.Optimize(f, x);
std::cout << "Minimum of squared function found with simulated annealing is " << x;
}

A few things, to get this working, but before that, just a disclaimer right from the codebase:

The given MatType must be arma::mat, arma::fmat, arma::sp_mat, or arma::sp_fmat, or it is not known to work! If you would like to try anyway, set the preprocessor macro ENS_DISABLE_TYPE_CHECKSbefore including ensmallen.hpp. However, you get to pick up all the pieces if there is a failure!

That said, you have to define ENS_DISABLE_TYPE_CHECKS see the code that I posted, you also have to update the Function signature to use arma::Mat<arma::uword> or arma::Mat<int>. There is also one line you have to change in the implementation of the SA optimizer, you have to change:

https://github.com/mlpack/ensmallen/blob/master/include/ensmallen_bits/sa/sa_impl.hpp#L101

to:

if (std::abs((float) energy - (float)oldEnergy) < tolerance)

I hope that helps, let me know if there is anything that is not clear.

[RichardPortelli]

That helps ! Thank you very much zoq. :) In general, if I have an optimisation problem with different variable types, for example integers and floats (to do hyperparameter optimization), which strategy/optimizer would you recommand?

[zoq]

ensmallen doesn't support mixed precision, so my recommendation is to convert the integer values to float, and at the end convert them back to integer. For some of the optimizers you can set individual parameter bounds as well (PSO, CMAES, etc.), in case you want to restrict the search space.

[RichardPortelli]

Thank you zoq :) Apologies if I abuse your kindness, I would like to ask a last question if I may. Does ensmallen support Bayesian optimization? I have seen there is a BBSMOEAD but I am not sure if it is the classical Bayesian implementation.

[zoq]

No direct Bayesian optimizer, but black-box optimizers, like CMAES, PSO, or MOEA/D-DE .

[RichardPortelli]

Thank you :)