Trust region with mixed search spaces

[khurram-ghani]

Related issue(s)/PRs: None

Summary

Trust regions with mixed search spaces are implemented by introducing the concept of a "product" trust region (and matching rule). This is analogous to TaggedProductSearchSpace.

TaggedProductSearchSpace internally contains a list of SearchSpace spaces; similarly UpdatableTrustRegionProduct contains a list of UpdatableTrustRegion regions.

The mixed_search_spaces notebook has been updated to show usage, and unit and integration tests have been added.

Integration tests do not exist for mixed search spaces (trust region or otherwise), mainly because we do not have a synthetic problem. The notebook creates a problem by discretizing one of the Branin dimensions. We could move this out of the notebook to create integration tests, or alternatively treat the notebook as "integration testing" for now.

Fully backwards compatible: yes

PR checklist

• [X] The quality checks are all passing
• [X] The bug case / new feature is covered by tests
• [X] Any new features are well-documented (in docstrings or notebooks)

[hstojic]

ok, now I have seen you have mentioned the integration tests in the description... I think we should definitely have them, I wouldn't rely on the notebook

[khurram-ghani]

Integration test added.