search

EmuKit / emukit

A Python-based toolbox of various methods in decision making, uncertainty quantification and statistical emulation: multi-fidelity, experimental design, Bayesian optimisation, Bayesian quadrature, etc.
https://emukit.github.io/emukit/
Apache License 2.0
605 stars 128 forks source link

NumPy DeprecationWarnings abound #302

Closed ekalosak closed 2 years ago

ekalosak commented 4 years ago

Anecdotally, passing pytests (272 passed tests) come with 26,000 DeprecationWarnings, mostly related to numpy issues.

There are some other errors wrt overflow, divby0, etc, but those are likely unrelated consequences of numerical issues in optimizing hyperparameters and acquisition functions.

apaleyes commented 2 years ago

Finally started looking into this. Let's document current state of affairs:

=============================================================================================== warnings summary ===============================================================================================
emukit/test_acquisitions.py: 195 warnings
emukit/bayesian_optimization/test_bayesian_optimization_loop.py: 10 warnings
emukit/bayesian_optimization/test_constrained_loop.py: 10 warnings
emukit/bayesian_optimization/test_cost_sensitive_bayesian_optimization.py: 10 warnings
emukit/bayesian_optimization/test_entropy_search.py: 10 warnings
emukit/bayesian_optimization/test_local_penalization_calculator.py: 5 warnings
emukit/bayesian_optimization/test_max_value_entropy_search.py: 5 warnings
emukit/bayesian_optimization/test_multipoint_expected_improvement.py: 5 warnings
emukit/core/test_outer_loop.py: 5 warnings
emukit/experimental_design/test_batch_experimental_design.py: 5 warnings
emukit/experimental_design/test_experimental_design_loop.py: 10 warnings
emukit/experimental_design/test_integrated_variance.py: 5 warnings
emukit/multi_fidelity/test_kernels.py: 94 warnings
emukit/multi_fidelity/test_models.py: 102 warnings
emukit/multi_fidelity/test_non_linear_models.py: 6930 warnings
emukit/quadrature/test_bounded_bq.py: 105 warnings
emukit/quadrature/test_quadrature_acquisitions.py: 40 warnings
emukit/quadrature/test_rbf_quadrature_kernel.py: 26 warnings
emukit/quadrature/test_vanilla_bq.py: 15 warnings
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/paramz/parameterized.py:271: DeprecationWarning:Assigning the 'data' attribute is an inherently unsafe operation and will be removed in the future.

emukit/test_acquisitions.py: 195 warnings
emukit/bayesian_optimization/test_bayesian_optimization_loop.py: 10 warnings
emukit/bayesian_optimization/test_constrained_loop.py: 10 warnings
emukit/bayesian_optimization/test_cost_sensitive_bayesian_optimization.py: 10 warnings
emukit/bayesian_optimization/test_entropy_search.py: 10 warnings
emukit/bayesian_optimization/test_local_penalization_calculator.py: 5 warnings
emukit/bayesian_optimization/test_max_value_entropy_search.py: 5 warnings
emukit/bayesian_optimization/test_multipoint_expected_improvement.py: 5 warnings
emukit/core/test_outer_loop.py: 5 warnings
emukit/experimental_design/test_batch_experimental_design.py: 5 warnings
emukit/experimental_design/test_experimental_design_loop.py: 10 warnings
emukit/experimental_design/test_integrated_variance.py: 5 warnings
emukit/multi_fidelity/test_kernels.py: 94 warnings
emukit/multi_fidelity/test_models.py: 102 warnings
emukit/multi_fidelity/test_non_linear_models.py: 6930 warnings
emukit/quadrature/test_bounded_bq.py: 105 warnings
emukit/quadrature/test_quadrature_acquisitions.py: 40 warnings
emukit/quadrature/test_rbf_quadrature_kernel.py: 26 warnings
emukit/quadrature/test_vanilla_bq.py: 15 warnings
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/paramz/parameterized.py:272: DeprecationWarning:Assigning the 'data' attribute is an inherently unsafe operation and will be removed in the future.

emukit/test_acquisitions.py: 117 warnings
emukit/bayesian_optimization/test_bayesian_optimization_loop.py: 6 warnings
emukit/bayesian_optimization/test_constrained_loop.py: 6 warnings
emukit/bayesian_optimization/test_cost_sensitive_bayesian_optimization.py: 6 warnings
emukit/bayesian_optimization/test_entropy_search.py: 6 warnings
emukit/bayesian_optimization/test_local_penalization_calculator.py: 3 warnings
emukit/bayesian_optimization/test_max_value_entropy_search.py: 3 warnings
emukit/bayesian_optimization/test_multipoint_expected_improvement.py: 3 warnings
emukit/core/test_outer_loop.py: 3 warnings
emukit/experimental_design/test_batch_experimental_design.py: 3 warnings
emukit/experimental_design/test_experimental_design_loop.py: 6 warnings
emukit/experimental_design/test_integrated_variance.py: 3 warnings
emukit/multi_fidelity/test_kernels.py: 135 warnings
emukit/multi_fidelity/test_models.py: 116 warnings
emukit/multi_fidelity/test_non_linear_models.py: 4900 warnings
emukit/quadrature/test_bounded_bq.py: 63 warnings
emukit/quadrature/test_quadrature_acquisitions.py: 24 warnings
emukit/quadrature/test_vanilla_bq.py: 9 warnings
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/paramz/core/parameter_core.py:290: DeprecationWarning:Assigning the 'data' attribute is an inherently unsafe operation and will be removed in the future.

emukit/test_acquisitions.py: 117 warnings
emukit/bayesian_optimization/test_bayesian_optimization_loop.py: 6 warnings
emukit/bayesian_optimization/test_constrained_loop.py: 6 warnings
emukit/bayesian_optimization/test_cost_sensitive_bayesian_optimization.py: 6 warnings
emukit/bayesian_optimization/test_entropy_search.py: 6 warnings
emukit/bayesian_optimization/test_local_penalization_calculator.py: 3 warnings
emukit/bayesian_optimization/test_max_value_entropy_search.py: 3 warnings
emukit/bayesian_optimization/test_multipoint_expected_improvement.py: 3 warnings
emukit/core/test_outer_loop.py: 3 warnings
emukit/experimental_design/test_batch_experimental_design.py: 3 warnings
emukit/experimental_design/test_experimental_design_loop.py: 6 warnings
emukit/experimental_design/test_integrated_variance.py: 3 warnings
emukit/multi_fidelity/test_kernels.py: 135 warnings
emukit/multi_fidelity/test_models.py: 116 warnings
emukit/multi_fidelity/test_non_linear_models.py: 4900 warnings
emukit/quadrature/test_bounded_bq.py: 63 warnings
emukit/quadrature/test_quadrature_acquisitions.py: 24 warnings
emukit/quadrature/test_vanilla_bq.py: 9 warnings
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/paramz/core/parameter_core.py:291: DeprecationWarning:Assigning the 'data' attribute is an inherently unsafe operation and will be removed in the future.

emukit/test_acquisitions.py: 39 warnings
emukit/bayesian_optimization/test_entropy_search.py: 2 warnings
emukit/bayesian_optimization/test_max_value_entropy_search.py: 1 warning
emukit/experimental_design/test_integrated_variance.py: 1 warning
emukit/multi_fidelity/test_kernels.py: 10 warnings
emukit/multi_fidelity/test_models.py: 20 warnings
emukit/multi_fidelity/test_non_linear_models.py: 12 warnings
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/GPy/core/__init__.py:80: DeprecationWarning:`np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

emukit/bayesian_optimization/test_cost_sensitive_bayesian_optimization.py::test_cost_sensitive_bayesian_optimization_loop
emukit/core/test_outer_loop.py::test_iteration_end_event
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/paramz/transformations.py:111: RuntimeWarning:overflow encountered in expm1

emukit/bayesian_optimization/test_local_penalization.py::test_local_penalization_at_batch_point
   /home/ubuntu/projects/emukit_org/emukit/emukit/bayesian_optimization/acquisitions/local_penalization.py:100: RuntimeWarning:divide by zero encountered in true_divide

emukit/core/test_parameter_space.py::test_check_in_domain_with_only_bandit_parameters
emukit/core/test_parameter_space.py::test_check_in_domain_with_bandit_parameter
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/numpy/core/fromnumeric.py:86: VisibleDeprecationWarning:Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.

emukit/core/optimization/test_local_search_acquisition_optimizer.py: 204 warnings
   /home/ubuntu/projects/emukit_org/emukit/emukit/core/optimization/local_search_acquisition_optimizer.py:146: DeprecationWarning:np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead

emukit/core/optimization/test_local_search_acquisition_optimizer.py::test_local_search_acquisition_optimizer_neighbours
emukit/core/optimization/test_local_search_acquisition_optimizer.py::test_local_search_acquisition_optimizer_neighbours
emukit/core/optimization/test_local_search_acquisition_optimizer.py::test_local_search_acquisition_optimizer_neighbours
emukit/core/optimization/test_local_search_acquisition_optimizer.py::test_local_search_acquisition_optimizer_neighbours
emukit/core/optimization/test_local_search_acquisition_optimizer.py::test_local_search_acquisition_optimizer_neighbours
emukit/core/optimization/test_local_search_acquisition_optimizer.py::test_local_search_acquisition_optimizer_neighbours
   /home/ubuntu/projects/emukit_org/emukit/emukit/core/optimization/local_search_acquisition_optimizer.py:92: DeprecationWarning:np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead

emukit/core/optimization/test_local_search_acquisition_optimizer.py::test_local_search_acquisition_optimizer_neighbours
emukit/core/optimization/test_local_search_acquisition_optimizer.py::test_local_search_acquisition_optimizer_neighbours
   /home/ubuntu/projects/emukit_org/emukit/emukit/core/optimization/local_search_acquisition_optimizer.py:102: DeprecationWarning:np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead

emukit/experimental_design/test_experimental_design_loop.py::test_loop
emukit/multi_fidelity/test_non_linear_models.py::TestNonLinearModel::test_optimize
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/GPy/kern/src/rbf.py:52: RuntimeWarning:overflow encountered in square

emukit/multi_fidelity/test_non_linear_models.py::TestNonLinearModel::test_optimize
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/GPy/kern/src/stationary.py:168: RuntimeWarning:overflow encountered in true_divide

emukit/multi_fidelity/test_non_linear_models.py::TestNonLinearModel::test_optimize
   /home/ubuntu/projects/emukit_org/emukit/.venv/lib/python3.8/site-packages/GPy/kern/src/rbf.py:76: RuntimeWarning:invalid value encountered in multiply

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================================ 317 passed, 26304 warnings in 87.12s (0:01:27) ================================================================================
apaleyes commented 2 years ago

So it looks that the absolute majority of these warnings is caused by code in https://github.com/sods/paramz and https://github.com/SheffieldML/GPy . PR above fixes those that our own code was triggering.

apaleyes commented 2 years ago

Suppressing these warnings is probably not a good idea though, as they are very general DeprecationWarning. If we suppress that, we might miss out others too. So I am inclined to leave things as they are now, after the PR is merged