search

mlflow / mlflow

Open source platform for the machine learning lifecycle
https://mlflow.org
Apache License 2.0
17.96k stars 4.05k forks source link

[BUG] `pytest --verbose tests/autologging --large` is flaky #4269

Closed harupy closed 3 years ago

harupy commented 3 years ago

pytest --verbose tests/autologging --large is flaky:

https://github.com/mlflow/mlflow/runs/2389644344#step:6:2031

An error occurs at the teardown of test_is_testing_respects_environment_variable in test_autologging_safety_unit.py.

Full error log:

pytest --verbose tests/autologging --large
============================= test session starts ==============================
platform linux -- Python 3.6.13, pytest-3.2.1, py-1.10.0, pluggy-0.4.0 -- /usr/share/miniconda/envs/test-environment/bin/python
cachedir: .cache
rootdir: /home/runner/work/mlflow/mlflow, inifile: pytest.ini
plugins: localserver-0.5.0, cov-2.6.0
collecting ... collected 150 items

tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration0] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration1] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration2] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration3] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration4] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration5] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration6] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration7] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration8] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration9] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration0] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration1] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration2] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration3] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration4] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration5] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration6] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration7] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration8] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_use_safe_patch_for_monkey_patching[integration9] PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autolog_respects_exclusive_flag PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autolog_respects_disable_flag PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autolog_respects_disable_flag_across_import_orders PASSED
tests/autologging/test_autologging_behaviors_integration.py::test_autolog_respects_silent_mode PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_autologging_warnings_are_redirected_as_expected PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_autologging_event_logging_and_warnings_respect_silent_mode PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_silent_mode_is_respected_in_multithreaded_environments PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_silent_mode_restores_warning_and_event_logging_behavior_correctly_if_errors_occur PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_silent_mode_operates_independently_across_integrations PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_silent_mode_and_warning_rerouting_respect_disabled_flag[False-False] PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_silent_mode_and_warning_rerouting_respect_disabled_flag[False-True] PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_silent_mode_and_warning_rerouting_respect_disabled_flag[True-False] PASSED
tests/autologging/test_autologging_behaviors_unit.py::test_silent_mode_and_warning_rerouting_respect_disabled_flag[True-True] PASSED
tests/autologging/test_autologging_safety_unit.py::test_is_testing_respects_environment_variable PASSED
🔥 tests/autologging/test_autologging_safety_unit.py::test_is_testing_respects_environment_variable ERROR
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_forwards_expected_arguments_to_function_based_patch_implementation PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_forwards_expected_arguments_to_class_based_patch PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_provides_expected_original_function PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_provides_expected_original_function_to_class_based_patch PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_propagates_exceptions_raised_from_original_function PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_logs_exceptions_raised_outside_of_original_function_as_warnings PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_propagates_exceptions_raised_outside_of_original_function_in_test_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_calls_original_function_when_patch_preamble_throws PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_returns_original_result_without_second_call_when_patch_postamble_throws PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_respects_disable_flag PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_returns_original_result_and_ignores_patch_return_value PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_validates_arguments_to_original_function_in_test_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_throws_when_autologging_runs_are_leaked_in_test_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_does_not_throw_when_autologging_runs_are_leaked_in_standard_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_validates_autologging_runs_when_necessary_in_test_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_does_not_validate_autologging_runs_in_standard_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_manages_run_if_specified_and_sets_expected_run_tags PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_does_not_manage_run_if_unspecified PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_preserves_signature_of_patched_function PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_provides_original_function_with_expected_signature PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_makes_expected_event_logging_calls_for_successful_patch_invocation PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_makes_expected_event_logging_calls_when_patch_implementation_throws PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_makes_expected_event_logging_calls_when_original_function_throws PASSED
tests/autologging/test_autologging_safety_unit.py::test_safe_patch_succeeds_when_event_logging_throws_in_standard_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_exception_safe_function_exhibits_expected_behavior_in_standard_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_exception_safe_function_exhibits_expected_behavior_in_test_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_exception_safe_class_exhibits_expected_behavior_in_standard_mode[object-_ExceptionSafeClass] PASSED
tests/autologging/test_autologging_safety_unit.py::test_exception_safe_class_exhibits_expected_behavior_in_standard_mode[ABC-_ExceptionSafeClass] PASSED
tests/autologging/test_autologging_safety_unit.py::test_exception_safe_class_exhibits_expected_behavior_in_test_mode[object-_ExceptionSafeClass] PASSED
tests/autologging/test_autologging_safety_unit.py::test_exception_safe_class_exhibits_expected_behavior_in_test_mode[ABC-_ExceptionSafeClass] PASSED
tests/autologging/test_autologging_safety_unit.py::test_patch_function_class_call_invokes_implementation_and_returns_result PASSED
tests/autologging/test_autologging_safety_unit.py::test_patch_function_class_call_handles_exceptions_properly[Exception] PASSED
tests/autologging/test_autologging_safety_unit.py::test_patch_function_class_call_handles_exceptions_properly[KeyboardInterrupt] PASSED
tests/autologging/test_autologging_safety_unit.py::test_with_managed_runs_yields_functions_and_classes_as_expected PASSED
tests/autologging/test_autologging_safety_unit.py::test_with_managed_run_with_non_throwing_function_exhibits_expected_behavior PASSED
tests/autologging/test_autologging_safety_unit.py::test_with_managed_run_with_throwing_function_exhibits_expected_behavior PASSED
tests/autologging/test_autologging_safety_unit.py::test_with_managed_run_with_non_throwing_class_exhibits_expected_behavior PASSED
tests/autologging/test_autologging_safety_unit.py::test_with_managed_run_with_throwing_class_exhibits_expected_behavior PASSED
tests/autologging/test_autologging_safety_unit.py::test_with_managed_run_sets_specified_run_tags PASSED
tests/autologging/test_autologging_safety_unit.py::test_with_managed_run_ends_run_on_keyboard_interrupt PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_args_succeeds_when_arg_sets_are_equivalent_or_identical PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_args_throws_when_extra_args_are_not_functions_classes_or_lists PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_args_throws_when_extra_args_are_not_exception_safe PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_args_succeeds_when_extra_args_are_exception_safe_functions_or_classes[object-_ExceptionSafeClass] PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_args_succeeds_when_extra_args_are_exception_safe_functions_or_classes[ABC-_ExceptionSafeClass] PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_args_throws_when_args_are_omitted PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_args_throws_when_arg_types_or_values_are_changed PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_autologging_run_validates_autologging_tag_correctly PASSED
tests/autologging/test_autologging_safety_unit.py::test_validate_autologging_run_validates_run_status_correctly PASSED
tests/autologging/test_autologging_safety_unit.py::test_try_mlflow_log_emits_exceptions_as_warnings_in_standard_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_try_mlflow_log_propagates_exceptions_in_test_mode PASSED
tests/autologging/test_autologging_safety_unit.py::test_session_manager_creates_session_before_patch_executes PASSED
tests/autologging/test_autologging_safety_unit.py::test_session_manager_exits_session_after_patch_executes PASSED
tests/autologging/test_autologging_safety_unit.py::test_session_manager_exits_session_if_error_in_patch PASSED
tests/autologging/test_autologging_safety_unit.py::test_session_manager_terminates_session_when_appropriate PASSED
tests/autologging/test_autologging_safety_unit.py::test_original_fn_runs_if_patch_should_not_be_applied PASSED
tests/autologging/test_autologging_safety_unit.py::test_patch_runs_if_patch_should_be_applied PASSED
tests/autologging/test_autologging_utils.py::test_log_fn_args_as_params[args0-kwargs0-expected0] PASSED
tests/autologging/test_autologging_utils.py::test_log_fn_args_as_params[args1-kwargs1-expected1] PASSED
tests/autologging/test_autologging_utils.py::test_log_fn_args_as_params[args2-kwargs2-expected2] PASSED
tests/autologging/test_autologging_utils.py::test_log_fn_args_as_params[args3-kwargs3-expected3] PASSED
tests/autologging/test_autologging_utils.py::test_log_fn_args_as_params[args4-kwargs4-expected4] PASSED
tests/autologging/test_autologging_utils.py::test_log_fn_args_as_params[args5-kwargs5-expected5] PASSED
tests/autologging/test_autologging_utils.py::test_log_fn_args_as_params_ignores_unwanted_parameters PASSED
tests/autologging/test_autologging_utils.py::test_wrap_patch_with_class PASSED
tests/autologging/test_autologging_utils.py::test_wrap_patch_with_module PASSED
tests/autologging/test_autologging_utils.py::test_if_getting_input_example_fails PASSED
tests/autologging/test_autologging_utils.py::test_if_model_signature_inference_fails PASSED
tests/autologging/test_autologging_utils.py::test_happy_path_works PASSED
tests/autologging/test_autologging_utils.py::test_avoids_collecting_input_example_if_not_needed PASSED
tests/autologging/test_autologging_utils.py::test_avoids_inferring_signature_if_not_needed PASSED
tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_logs_all_metrics PASSED
tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_flush_logs_to_mlflow PASSED
tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_runs_training_and_logging_in_correct_ratio PASSED
tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_chunks_metrics_when_batch_logging PASSED
tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_records_time_correctly PASSED
tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_logs_timestamps_as_int_milliseconds PASSED
tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_continues_if_log_batch_fails PASSED
tests/autologging/test_autologging_utils.py::test_autologging_integration_calls_underlying_function_correctly PASSED
tests/autologging/test_autologging_utils.py::test_autologging_integration_stores_and_updates_config PASSED
tests/autologging/test_autologging_utils.py::test_autologging_integration_forwards_positional_and_keyword_arguments_as_expected PASSED
tests/autologging/test_autologging_utils.py::test_autologging_integration_validates_structure_of_autolog_function PASSED
tests/autologging/test_autologging_utils.py::test_autologging_integration_makes_expected_event_logging_calls PASSED
tests/autologging/test_autologging_utils.py::test_autologging_integration_succeeds_when_event_logging_throws_in_standard_mode PASSED
tests/autologging/test_autologging_utils.py::test_get_autologging_config_returns_configured_values_or_defaults_as_expected PASSED
tests/autologging/test_autologging_utils.py::test_autologging_is_disabled_returns_expected_values PASSED
tests/autologging/test_autologging_utils.py::test_autologging_event_logger_default_implementation_does_not_throw_for_valid_inputs PASSED
tests/autologging/test_autologging_utils.py::test_autologging_event_logger_default_impl_warns_for_log_autolog_called_with_deprecated_args PASSED
tests/autologging/test_autologging_utils.py::test_check_version_in_range PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[fastai-1.0.60-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[fastai-1.0.50-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[gluon-1.6.1-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[gluon-1.5.0-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[keras-2.2.4-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[keras-2.2.3-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[lightgbm-2.3.1-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[lightgbm-2.3.0-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[statsmodels-0.11.1-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[statsmodels-0.11.0-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[tensorflow-1.15.4-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[tensorflow-1.15.3-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[xgboost-0.90-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[xgboost-0.89-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[sklearn-0.20.3-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[sklearn-0.20.2-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[pytorch-1.0.5-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[pytorch-1.0.4-False] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[pyspark.ml-3.1.0-True] PASSED
tests/autologging/test_autologging_utils.py::test_is_autologging_integration_supported[pyspark.ml-3.0.0-False] PASSED
tests/autologging/test_autologging_utils.py::test_disable_for_unsupported_versions_warning_sklearn_integration PASSED
tests/autologging/test_training_session.py::test_should_log_always_returns_true_in_root_session[True] PASSED
tests/autologging/test_training_session.py::test_should_log_always_returns_true_in_root_session[False] PASSED
tests/autologging/test_training_session.py::test_nested_sessions[True] PASSED
tests/autologging/test_training_session.py::test_nested_sessions[False] PASSED
tests/autologging/test_training_session.py::test_parent_session_overrides_child_session PASSED
tests/autologging/test_training_session.py::test_should_log_returns_false_when_parrent_session_has_the_same_class PASSED

=========================== slowest 5 test durations ===========================
9.45s call     tests/autologging/test_autologging_behaviors_integration.py::test_autolog_respects_silent_mode
8.91s call     tests/autologging/test_autologging_behaviors_integration.py::test_autolog_respects_exclusive_flag
4.04s call     tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_records_time_correctly
3.72s setup    tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration0]
2.51s call     tests/autologging/test_autologging_behaviors_integration.py::test_autologging_integrations_expose_configs_and_support_disablement[integration0]
==================================== ERRORS ====================================
______ ERROR at teardown of test_is_testing_respects_environment_variable ______

    @pytest.fixture(autouse=True)
    def clean_up_leaked_runs():
        """
        Certain test cases validate safety API behavior when runs are leaked. Leaked runs that
        are not cleaned up between test cases may result in cascading failures that are hard to
        debug. Accordingly, this fixture attempts to end any active runs it encounters and
        throws an exception (which reported as an additional error in the pytest execution output).
        """
        try:
            yield
>           assert not mlflow.active_run(), "test case unexpectedly leaked a run!"
E           AssertionError: test case unexpectedly leaked a run!
E           assert not <ActiveRun: >
E            +  where <ActiveRun: > = <function active_run at 0x7fb66b381f28>()
E            +    where <function active_run at 0x7fb66b381f28> = mlflow.active_run

tests/autologging/test_autologging_safety_unit.py:80: AssertionError

During handling of the above exception, another exception occurred:

    @pytest.fixture(autouse=True)
    def clean_up_leaked_runs():
        """
        Certain test cases validate safety API behavior when runs are leaked. Leaked runs that
        are not cleaned up between test cases may result in cascading failures that are hard to
        debug. Accordingly, this fixture attempts to end any active runs it encounters and
        throws an exception (which reported as an additional error in the pytest execution output).
        """
        try:
            yield
            assert not mlflow.active_run(), "test case unexpectedly leaked a run!"
        finally:
            while mlflow.active_run():
>               mlflow.end_run()

tests/autologging/test_autologging_safety_unit.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
mlflow/tracking/fluent.py:277: in end_run
    MlflowClient().set_terminated(run.info.run_id, status)
mlflow/tracking/client.py:1373: in set_terminated
    self._tracking_client.set_terminated(run_id, status, end_time)
mlflow/tracking/_tracking_service/client.py:326: in set_terminated
    run_id, run_status=RunStatus.from_string(status), end_time=end_time
mlflow/store/tracking/sqlalchemy_store.py:446: in update_run_info
    run = self._get_run(run_uuid=run_id, session=session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <mlflow.store.tracking.sqlalchemy_store.SqlAlchemyStore object at 0x7fb5688ee470>
session = <sqlalchemy.orm.session.Session object at 0x7fb56889b080>
run_uuid = '9dd96641c29a445aaa4e8ebc886ad61f', eager = False

    def _get_run(self, session, run_uuid, eager=False):
        """
            :param eager: If ``True``, eagerly loads the run's summary metrics (``latest_metrics``),
                          params, and tags when fetching the run. If ``False``, these attributes
                          are not eagerly loaded and will be loaded when their corresponding
                          object properties are accessed from the resulting ``SqlRun`` object.
            """
        query_options = self._get_eager_run_query_options() if eager else []
        runs = (
            session.query(SqlRun).options(*query_options).filter(SqlRun.run_uuid == run_uuid).all()
        )

        if len(runs) == 0:
            raise MlflowException(
>               "Run with id={} not found".format(run_uuid), RESOURCE_DOES_NOT_EXIST
            )
E           mlflow.exceptions.MlflowException: Run with id=9dd96641c29a445aaa4e8ebc886ad61f not found

eager      = False
query_options = []
run_uuid   = '9dd96641c29a445aaa4e8ebc886ad61f'
runs       = []
self       = <mlflow.store.tracking.sqlalchemy_store.SqlAlchemyStore object at 0x7fb5688ee470>
session    = <sqlalchemy.orm.session.Session object at 0x7fb56889b080>

mlflow/store/tracking/sqlalchemy_store.py:392: MlflowException
--------------------------- Captured stderr teardown ---------------------------
2021/04/20 11:06:28 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2021/04/20 11:06:28 INFO mlflow.store.db.utils: Updating database tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 451aebb31d03, add metric step
INFO  [alembic.runtime.migration] Running upgrade 451aebb31d03 -> 90e64c465722, migrate user column to tags
INFO  [alembic.runtime.migration] Running upgrade 90e64c465722 -> 181f10493468, allow nulls for metric values
INFO  [alembic.runtime.migration] Running upgrade 181f10493468 -> df50e92ffc5e, Add Experiment Tags Table
INFO  [alembic.runtime.migration] Running upgrade df50e92ffc5e -> 7ac759974ad8, Update run tags with larger limit
INFO  [alembic.runtime.migration] Running upgrade 7ac759974ad8 -> 89d4b8295536, create latest metrics table
INFO  [89d4b8295536_create_latest_metrics_table_py] Migration complete!
INFO  [alembic.runtime.migration] Running upgrade 89d4b8295536 -> 2b4d017a5e9b, add model registry tables to db
INFO  [2b4d017a5e9b_add_model_registry_tables_to_db_py] Adding registered_models and model_versions tables to database.
INFO  [2b4d017a5e9b_add_model_registry_tables_to_db_py] Migration complete!
INFO  [alembic.runtime.migration] Running upgrade 2b4d017a5e9b -> cfd24bdc0731, Update run status constraint with killed
INFO  [alembic.runtime.migration] Running upgrade cfd24bdc0731 -> 0a8213491aaa, drop_duplicate_killed_constraint
WARNI [0a8213491aaa_drop_duplicate_killed_constraint_py] Failed to drop check constraint. Dropping check constraints may not be supported by your SQL database. Exception content: No support for ALTER of constraints in SQLite dialectPlease refer to the batch mode feature which allows for SQLite migrations using a copy-and-move strategy.
INFO  [alembic.runtime.migration] Running upgrade 0a8213491aaa -> 728d730b5ebd, add registered model tags table
INFO  [alembic.runtime.migration] Running upgrade 728d730b5ebd -> 27a6a02d2cf1, add model version tags table
INFO  [alembic.runtime.migration] Running upgrade 27a6a02d2cf1 -> 84291f40a231, add run_link to model_version
INFO  [alembic.runtime.migration] Running upgrade 84291f40a231 -> a8c4a736bde6, allow nulls for run_id
INFO  [alembic.runtime.migration] Running upgrade a8c4a736bde6 -> 39d1c3be5f05, add_is_nan_constraint_for_metrics_tables_if_necessary
INFO  [alembic.runtime.migration] Running upgrade 39d1c3be5f05 -> c48cb773bb87, reset_default_value_for_is_nan_in_metrics_table_for_mysql
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
=============================== warnings summary ===============================
tests/autologging/test_autologging_utils.py::test_batch_metrics_logger_continues_if_log_batch_fails
  /home/runner/work/mlflow/mlflow/mlflow/utils/autologging_utils/__init__.py:202: UserWarning: Logging to MLflow failed: asdf
    try_mlflow_log(MlflowClient().log_batch, run_id=current_run_id, metrics=metrics_slice)

tests/autologging/test_autologging_utils.py::test_disable_for_unsupported_versions_warning_sklearn_integration
  /home/runner/work/mlflow/mlflow/mlflow/utils/autologging_utils/__init__.py:381: UserWarning: Autologging utilities may not work properly on scikit-learn < 0.20.3 (current version: 0.20.2)
    return _autolog(*args, **kwargs)
  /home/runner/work/mlflow/mlflow/mlflow/utils/autologging_utils/__init__.py:381: UserWarning: Autologging utilities may not work properly on scikit-learn < 0.20.3 (current version: 0.20.2)
    return _autolog(*args, **kwargs)
  /home/runner/work/mlflow/mlflow/mlflow/utils/autologging_utils/__init__.py:381: UserWarning: Autologging utilities may not work properly on scikit-learn < 0.20.3 (current version: 0.20.2)
    return _autolog(*args, **kwargs)
  /home/runner/work/mlflow/mlflow/mlflow/utils/autologging_utils/__init__.py:381: UserWarning: Autologging utilities may not work properly on scikit-learn < 0.20.3 (current version: 0.20.2)
    return _autolog(*args, **kwargs)

-- Docs: http://doc.pytest.org/en/latest/warnings.html
=============== 150 passed, 5 warnings, 1 error in 48.33 seconds ===============
harupy commented 3 years ago 
import mlflow

mlflow.set_tracking_uri("foo")

# A run leaks somewhere
mlflow.start_run()

# At setup for test_is_testing_respects_environment_variable,
# switch to a different tracking URI via `tracking_uri_mock`
mlflow.set_tracking_uri("bar")

# test_is_testing_respects_environment_variable passes

# At teardown for test_is_testing_respects_environment_variable:
assert not mlflow.active_run() -> fails since an active run exists
mlflow.end_run() -> fails since the active run doesn't belong to the tracking URI "bar"

I'm investigating:

harupy commented 3 years ago

@dbczumar Thanks for investigating!