Address many unexpected warnings in pytest

andrewlee94 commented 2 months ago

Partially Addresses #1363

Summary/Motivation:

This fixes a lot of unexpected warnings being emitted during pytest runs.

Changes proposed in this PR:

Fix instances of non-test classes with names TestX
Fix warnings about regex escapes
Remove a number of redundant tests for surrogate plotting. Also get plotting methods to return the plot objects so we can close them.
Remove some return statements in test methods/functions.

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the license terms described in the LICENSE.txt file at the top level of this directory.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

lbianchi-lbl commented 1 month ago

For reference, this should be a list of the remaining warnings from Pysmo:

idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test__init__08[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test__init__08[DataFrame]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test__init__09[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_predict_output_01[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_predict_output[array]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/kriging.py:354: RuntimeWarning: invalid value encountered in log
    log_like = (0.5 * ns * np.log(ssd)) + (0.5 * lndetcov)

idaes/core/surrogate/pysmo/tests/test_kriging.py: 314 warnings
idaes/core/surrogate/tests/test_pysmo_surrogate.py: 35 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/kriging.py:540: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    y_prediction[i, 0] = mean + np.matmul(

idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[DataFrame]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[DataFrame]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_kriging.py:346: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    y_prediction_exp[i, 0] = mean + np.matmul(

idaes/core/surrogate/pysmo/tests/test_kriging.py: 52 warnings
idaes/core/surrogate/tests/test_pysmo_surrogate.py: 1044 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/kriging.py:592: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    y_pred[i, 0] = self.optimal_mean + np.matmul(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test__init__21[array-array]
idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test__init__21[array-DataFrame]
idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test__init__21[DataFrame-array]
idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test__init__21[DataFrame-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/polynomial_regression.py:359: UserWarning: The maximum allowed polynomial order is 10. Value has been adjusted to 10.
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_results_generation_01[array-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py:1648: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.
    expected_df = pd.concat(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_results_generation_01[array-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/polynomial_regression.py:1003: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.
    results_df = pd.concat([results_df, pd.Series({"k": beta[0, 0]})], axis=0)

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_results_generation_02[array-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py:1690: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.
    expected_df = pd.concat(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_results_generation_03[array-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py:1728: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.
    expected_df = pd.concat(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_confint_regression_01
idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_confint_regression_02
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/polynomial_regression.py:1452: UserWarning: Polynomial regression generates poor fit for the dataset
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_basis_generation[array]
idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_basis_generation[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py:531: RuntimeWarning: divide by zero encountered in log
    expected_output_3 = np.nan_to_num(distance_array**2 * np.log(distance_array))

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_basis_generation[array]
idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_basis_generation[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py:531: RuntimeWarning: invalid value encountered in multiply
    expected_output_3 = np.nan_to_num(distance_array**2 * np.log(distance_array))

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py: 12 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/radial_basis_function.py:1108: UserWarning: The parameter matrix A in A.x=B is ill-conditioned (condition number > 1e10). The solution returned may be inaccurate or unstable - inspect rmse error. Regularization (if not already done) may improve solution
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_rbf_predict_output_06[array]
idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_rbf_predict_output_06[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py:2093: RuntimeWarning: divide by zero encountered in log
    np.nan_to_num(distance_vec**2 * np.log(distance_vec)), results.weights

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_rbf_predict_output_06[array]
idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_rbf_predict_output_06[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py:2093: RuntimeWarning: invalid value encountered in multiply
    np.nan_to_num(distance_vec**2 * np.log(distance_vec)), results.weights

idaes/core/surrogate/pysmo/tests/test_sampling.py: 10 warnings
idaes/core/surrogate/pysmo/tests/test_sampling_modified.py: 4 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:183: UserWarning: The returned number of samples is less than the requested number due to repetitions during nearest neighbour selection.
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_sampling.py::TestCVTSampling::test__init__selection_11[array]
idaes/core/surrogate/pysmo/tests/test_sampling.py::TestCVTSampling::test__init__selection_11[DataFrame]
idaes/core/surrogate/pysmo/tests/test_sampling.py::TestCVTSampling::test__init__creation_11[list]
idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::CVTSamplingTestCases::test__init__creation_11
idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::CVTSamplingTestCases::test__init__selection_13
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:1420: UserWarning: Tolerance too tight. CVT algorithm may take long time to converge.
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_sampling.py::TestCustomSampling::test_generate_from_dist_normal_enforced_gaussian_bounds
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:1787: UserWarning: Points adjusted to remain within specified Gaussian bounds. This may affect the underlying distribution.
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_sampling.py: 22 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:1799: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    var_values[k] = rep_value

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_08
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:391: UserWarning: The following columns were dropped: ['x2', 'y2']
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_09
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:391: UserWarning: The following columns were dropped: ['x2']
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_10
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:391: UserWarning: The following columns were dropped: ['x1', 'y2']
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_11
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:391: UserWarning: The following columns were dropped: ['y1']
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_21
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:451: UserWarning: The following columns were dropped: [2]
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_22
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:451: UserWarning: The following columns were dropped: [1, 3]
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_23
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:451: UserWarning: The following columns were dropped: [1]
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_24
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:451: UserWarning: The following columns were dropped: [0, 3]
    warnings.warn(warn_str)

idaes/core/surrogate/tests/test_pysmo_surrogate.py: 7824 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo_surrogate.py:516: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    outputs[i, j] = result.model.predict_output(row_data)

codecov-commenter commented 1 month ago

Codecov Report

Attention: Patch coverage is 81.81818% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 77.88%. Comparing base (bc2f6c0) to head (67a0b4f).

Files	Patch %	Lines
idaes/apps/matopt/opt/pyomo_modeling.py	0.00%	4 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1403 +/- ## ========================================== + Coverage 77.70% 77.88% +0.18% ========================================== Files 394 394 Lines 65049 65061 +12 Branches 14383 14383 ========================================== + Hits 50546 50675 +129 + Misses 11902 11797 -105 + Partials 2601 2589 -12 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

IDAES / idaes-pse