Run coverage run --source=mcfly -m pytest
============================= test session starts ==============================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/runner/work/mcfly/mcfly
plugins: cov-4.0.0
collected 52 items
tests/test_base_hyperparameter_generator.py ... [ 5%]
tests/test_cnn.py ....... [ 19%]
tests/test_deep_conv_lstm.py .... [ 26%]
tests/test_find_architecture.py .F................. [ 63%]
tests/test_inception_time.py ...... [ 75%]
tests/test_integration.py . [ 76%]
tests/test_modelgen.py ... [ 82%]
tests/test_resnet.py ...... [ 94%]
tests/test_storage.py ... [100%]
=================================== FAILURES ===================================
__ FindArchitectureBasicSuite.test_find_best_architecture_with_class_weights ___
self = <test_find_architecture.FindArchitectureBasicSuite testMethod=test_find_best_architecture_with_class_weights>
def test_find_best_architecture_with_class_weights(self):
"""Model should not ignore tiny class with huge class weight. Note that this test is non-deterministic,
even though a seed was set. Note2 that this test is very slow, taking up 40% of all mcfly test time."""
tf.random.set_seed(1234) # Needs tensorflow API v2
X_train, y_train = _create_2_class_labeled_dataset(1, 999) # very unbalanced
X_val, y_val = _create_2_class_labeled_dataset(1, 99)
X_test, y_test = _create_2_class_labeled_dataset(10, 10)
class_weight = {0: 2, 1: 0.002}
best_model, _, _, _ = find_architecture.find_best_architecture(
X_train, y_train, X_val, y_val, verbose=False, subset_size=1000,
number_of_models=5, nr_epochs=1, model_type='CNN', class_weight=class_weight)
probabilities = best_model.predict(X_test)
predicted = probabilities.argmax(axis=1)
> np.testing.assert_array_equal(predicted, y_test.argmax(axis=1))
E AssertionError:
E Arrays are not equal
E
E Mismatched elements: 10 / 20 (50%)
E Max absolute difference: 1
E Max relative difference: 0.
E x: array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
E y: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
tests/test_find_architecture.py:224: AssertionError
----------------------------- Captured stdout call -----------------------------
Set maximum kernel size for InceptionTime models to number of timesteps.
Set maximum kernel size for InceptionTime models to number of timesteps.
Generated models will be trained on subset of the data (subset size: 1000).
1/1 [==============================] - ETA: 0s
1/1 [==============================] - 1s 726ms/step
------------------------------ Captured log call -------------------------------
WARNING absl:optimizer.py:106 `lr` is deprecated, please use `learning_rate` instead, or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.Adam.
WARNING absl:optimizer.py:106 `lr` is deprecated, please use `learning_rate` instead, or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.Adam.
WARNING absl:optimizer.py:106 `lr` is deprecated, please use `learning_rate` instead, or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.Adam.
WARNING absl:optimizer.py:106 `lr` is deprecated, please use `learning_rate` instead, or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.Adam.
WARNING absl:optimizer.py:106 `lr` is deprecated, please use `learning_rate` instead, or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.Adam.
=============================== warnings summary ===============================
tests/test_base_hyperparameter_generator.py::test_regularization_is_float
/home/runner/work/mcfly/mcfly/tests/test_base_hyperparameter_generator.py:8: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
assert isinstance(reg, np.float), "Expected different type."
tests/test_find_architecture.py::FindArchitectureBasicSuite::test_find_best_architecture
tests/test_integration.py::IntegrationSuite::test_integration
tests/test_storage.py::StorageSuite::test_loadmodel
tests/test_storage.py::StorageSuite::test_savemodel
tests/test_storage.py::StorageSuite::test_savemodel_keras
/home/runner/work/mcfly/mcfly/mcfly/modelgen.py:86: UserWarning: Specified number_of_models is smaller than the given number of model types.
warnings.warn("Specified number_of_models is smaller than the given number of model types.")
tests/test_find_architecture.py::FindArchitectureBasicSuite::test_find_best_architecture
/home/runner/work/mcfly/mcfly/mcfly/find_architecture.py:382: UserWarning: Best model not better than kNN: [] vs 0.6666666666666666
warnings.warn('Best model not better than kNN: ' +
tests/test_find_architecture.py::FindArchitectureBasicSuite::test_find_best_architecture_with_class_weights
/home/runner/work/mcfly/mcfly/mcfly/find_architecture.py:382: UserWarning: Best model not better than kNN: [] vs 1.0
warnings.warn('Best model not better than kNN: ' +
tests/test_find_architecture.py::HistoryStoringSuite::test_store_train_history_as_json_contains_expected_attributes
tests/test_find_architecture.py::HistoryStoringSuite::test_store_train_history_as_json_contains_expected_attributes
tests/test_find_architecture.py::HistoryStoringSuite::test_store_train_history_as_json_metrics_is_dict
tests/test_find_architecture.py::HistoryStoringSuite::test_store_train_history_as_json_metrics_is_dict
/home/runner/work/mcfly/mcfly/tests/test_find_architecture.py:342: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
'val_loss': [np.float(1), np.float(1)], 'val_accuracy': [np.float64(0), np.float64(0)],
tests/test_storage.py::StorageSuite::test_loadmodel
tests/test_storage.py::StorageSuite::test_savemodel
/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/numpy/lib/npyio.py:521: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
arr = np.asanyarray(arr)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_find_architecture.py::FindArchitectureBasicSuite::test_find_best_architecture_with_class_weights - AssertionError:
Arrays are not equal
Mismatched elements: 10 / 20 (50%)
Max absolute difference: 1
Max relative difference: 0.
x: array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
y: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
============ 1 failed, 51 passed, 14 warnings in 237.59s (0:03:57) =============
Error: Process completed with exit code 1.
tests/test_find_architecture.py::FindArchitectureBasicSuite::test_find_best_architecture_with_class_weights
https://github.com/NLeSC/mcfly/actions/runs/3693703246/jobs/6254006520