Update Bayes search for NN modules

[bryanlimy]

• use Categorical to address https://github.com/alan-turing-institute/autoemulate/issues/199
• subsequently, we get following error due to deprecated reference to np.int in skopt. As a quick fix, I added np.int=np.int64 in hyperparam_searching.py. There is an open issue on this https://github.com/scikit-optimize/scikit-optimize/issues/1171 on skopt.

  autoemulate - Failed to perform hyperparameter search on NNMlp
  Traceback (most recent call last):
  File "/Users/bryanlimy/Git/autoemulate/autoemulate/hyperparam_searching.py", line 91, in _optimize_params
    searcher.fit(X, y)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/searchcv.py", line 466, in fit
    super().fit(X=X, y=y, groups=groups, **fit_params)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/sklearn/base.py", line 1474, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/sklearn/model_selection/_search.py", line 970, in fit
    self._run_search(evaluate_candidates)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/searchcv.py", line 512, in _run_search
    optim_result = self._step(
                   ^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/searchcv.py", line 400, in _step
    params = optimizer.ask(n_points=n_points)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/optimizer/optimizer.py", line 395, in ask
    x = opt.ask()
        ^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/optimizer/optimizer.py", line 367, in ask
    return self._ask()
           ^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/optimizer/optimizer.py", line 434, in _ask
    return self.space.rvs(random_state=self.rng)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/space/space.py", line 900, in rvs
    columns.append(dim.rvs(n_samples=n_samples, random_state=rng))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/space/space.py", line 698, in rvs
    return self.inverse_transform(list(choices))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/space/space.py", line 685, in inverse_transform
    inv_transform = super(Categorical, self).inverse_transform(Xt)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/space/space.py", line 168, in inverse_transform
    return self.transformer.inverse_transform(Xt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/space/transformers.py", line 309, in inverse_transform
    X = transformer.inverse_transform(X)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/skopt/space/transformers.py", line 275, in inverse_transform
    return np.round(X_orig).astype(np.int)
                                   ^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/autoemulate/lib/python3.11/site-packages/numpy/__init__.py", line 324, in __getattr__
    raise AttributeError(__former_attrs__[attr])
  AttributeError: module 'numpy' has no attribute 'int'.
  `np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
  The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

• Added LBFGS optimizer to NN modules to match with scikit-learn MLP, addressing https://github.com/alan-turing-institute/autoemulate/issues/192
• Updated initialize_optimizer to ignore weight_decay if LBFGS is used as it doesn't support weight decay in PyTorch.
• Lower the max learning rate for hyperparameter search to 1e-3 to avoid gradient exploding.
• Set verbose=0 and error_score=np.nan in _optimize_params so that the values to RandomizedSearchCV and BayesSearchCV are the same. The default value of error_score to RandomizedSearchCV is np.nan which is different from the default value of "raise" inBayesSearchCV`.

[github-actions[bot]]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
autoemulate
hyperparam_searching.py
autoemulate/emulators
neural_net_torch.py
autoemulate/emulators/neural_networks
mlp.py
rbf.py					332-336
tests
test_torch.py
Project Total

This report was generated by python-coverage-comment-action

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 91.66667% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 90.83%. Comparing base (b512fe3) to head (59ee578).

Files	Patch %	Lines
autoemulate/emulators/neural_networks/rbf.py	33.33%	2 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #205 +/- ## ========================================== + Coverage 90.61% 90.83% +0.22% ========================================== Files 45 45 Lines 2269 2281 +12 ========================================== + Hits 2056 2072 +16 + Misses 213 209 -4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[bryanlimy]

@mastoffel I think it is unavoidable that some of the MLP search cases would return NaN, especially with adam optimizer

[mastoffel]

@mastoffel I think it is unavoidable that some of the MLP search cases would return NaN, especially with adam optimizer

@bryanlimy Ah, I think for me it seems to come from sgd, when I remove it I don't really get NaN warnings. Is that different for you?

[bryanlimy]

@mastoffel I think it is unavoidable that some of the MLP search cases would return NaN, especially with adam optimizer

@bryanlimy Ah, I think for me it seems to come from sgd, when I remove it I don't really get NaN warnings. Is that different for you?

Sorry yes I meant to say SGD. Same issue in https://github.com/alan-turing-institute/autoemulate/issues/191#issuecomment-1967106309

[bryanlimy]

@mastoffel setting a higher minimum learning rate seems to be ok for now. I found a number of Issues where users report ValueError: Input y contains NaN. with BayesSearchCV, I will need to check if it is cause by something else in our code. Ideally we would want the searcher to move on to the next settings if the current model report NaN.

[mastoffel]

@mastoffel setting a higher minimum learning rate seems to be ok for now. I found a number of Issues where users report ValueError: Input y contains NaN. with BayesSearchCV, I will need to check if it is cause by something else in our code. Ideally we would want the searcher to move on to the next settings if the current model report NaN.

@bryanlimy

• It's definitely better, but I'm still get NaN's regularly with SGD, so I do suggest we take it out for now. Need to figure out a way to catch these warnings when running in parallel in the future though.
• BayesSearchCV seems to work now. Was that related to the learning rate too?

[mastoffel]

@bryanlimy ok, seems to work now!

• I'll open an issue about the NaN's warning, I think we just need to catch that.
• NNMlp still seems to not do as well as the sklearn MLP. I'm currently not sure why that is, but there are a few potential things to change here, but I think we should just put that into a different issue for now.
• Feel free to merge and squash commits!