test_tensorboard_callback failures

[opoplawski]

System information OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Fedora rawhide Sklearn-genetic-opt version: 0.10.1 Scikit-learn version: 1.2.2 Python version: 3.11.4

Describe the bug When trying to build the Fedora package:

+ /usr/bin/pytest --ignore sklearn_genetic/tests/test_mlflow.py
=========================================================================================================== test session starts ============================================================================================================
platform linux -- Python 3.11.4, pytest-7.3.2, pluggy-1.0.0
rootdir: /home/orion/fedora/python-sklearn-genetic-opt/Sklearn-genetic-opt-0.10.1
plugins: xprocess-0.22.2, remotedata-0.3.3, asyncio-0.21.0, libtmux-0.21.0, forked-1.6.0, mock-3.10.0, cov-4.0.0, flake8-1.1.1, timeout-2.1.0, rerunfailures-11.0, pyfakefs-5.2.2, xdist-3.3.1, doctestplus-0.12.0, openfiles-0.5.0, anyio-3.5.0, mpi-0.6, hypothesis-6.62.1, localserver-0.7.0, arraydiff-0.5.0
asyncio: mode=Mode.STRICT
collected 117 items                                                                                                                                                                                                                        

sklearn_genetic/callbacks/tests/test_callbacks.py ..............FFFF                                                                                                                                                                 [ 15%]
sklearn_genetic/schedules/tests/test_schedules.py ...............                                                                                                                                                                    [ 28%]
sklearn_genetic/space/tests/test_space.py ..................                                                                                                                                                                         [ 43%]
sklearn_genetic/tests/test_feature_selection.py .............................                                                                                                                                                        [ 68%]
sklearn_genetic/tests/test_genetic_search.py .................................                                                                                                                                                       [ 96%]
sklearn_genetic/tests/test_plots.py ...                                                                                                                                                                                              [ 99%]
sklearn_genetic/tests/test_utils.py .                                                                                                                                                                                                [100%]

================================================================================================================= FAILURES =================================================================================================================
_______________________________________________________________________________________________ test_tensorboard_callback[callback0-./logs] ________________________________________________________________________________________________

callback = <sklearn_genetic.callbacks.loggers.TensorBoard object at 0x7f3fc04c5490>, path = './logs'

    @pytest.mark.parametrize(
        "callback, path",
        [
            (TensorBoard(), "./logs"),
            (TensorBoard(log_dir="./sklearn_logs"), "./sklearn_logs"),
            (TensorBoard(log_dir="./logs", run_id="0"), "./logs/0"),
            (TensorBoard(log_dir="./logs", run_id="1"), "./logs/1"),
        ],
    )
    def test_tensorboard_callback(callback, path):
        assert check_callback(callback) == [callback]

        clf = DecisionTreeClassifier()
        evolved_estimator = GASearchCV(
            clf,
            cv=3,
            scoring="accuracy",
            generations=2,
            param_grid={
                "min_weight_fraction_leaf": Continuous(0, 0.5),
                "max_depth": Integer(2, 20),
                "max_leaf_nodes": Integer(2, 30),
            },
            verbose=False,
        )

>       evolved_estimator.fit(X_train, y_train, callbacks=callback)

sklearn_genetic/callbacks/tests/test_callbacks.py:239: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn_genetic/genetic_search.py:492: in fit
    pop, log, n_gen = self._select_algorithm(pop=self._pop, stats=self._stats, hof=self._hof)
sklearn_genetic/genetic_search.py:572: in _select_algorithm
    pop, log, gen = selected_algorithm(
sklearn_genetic/algorithms.py:305: in eaMuPlusLambda
    if eval_callbacks(**callbacks_step_args):
sklearn_genetic/callbacks/validations.py:66: in eval_callbacks
    decision = callback_method(record, logbook, estimator)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sklearn_genetic.callbacks.loggers.TensorBoard object at 0x7f3fc04c5490>, record = {'fitness': 0.3655195344970906, 'fitness_max': 0.7564422277639234, 'fitness_min': 0.19035743973399835, 'fitness_std': 0.186707543836711}
logbook = [{'gen': 0, 'nevals': 50, 'fitness': 0.3655195344970906, 'fitness_std': 0.186707543836711, 'fitness_max': 0.7564422277639234, 'fitness_min': 0.19035743973399835}]
estimator = GASearchCV(estimator=DecisionTreeClassifier(), generations=2,
           param_grid={'max_depth': <sklearn_genetic.spa...ace.space.Continuous object at 0x7f3fbc1bcd90>},
           return_train_score=True, scoring='accuracy', verbose=False)

    def on_step(self, record=None, logbook=None, estimator=None):
        # Get the last metric value
        stats = logbook[-1]

        # Create logs files placeholder
>       writer = tf.summary.create_file_writer(self.path)
E       AttributeError: 'NoneType' object has no attribute 'summary'

sklearn_genetic/callbacks/loggers.py:112: AttributeError
___________________________________________________________________________________________ test_tensorboard_callback[callback1-./sklearn_logs] ____________________________________________________________________________________________

callback = <sklearn_genetic.callbacks.loggers.TensorBoard object at 0x7f3fc035ab90>, path = './sklearn_logs'

    @pytest.mark.parametrize(
        "callback, path",
        [
            (TensorBoard(), "./logs"),
            (TensorBoard(log_dir="./sklearn_logs"), "./sklearn_logs"),
            (TensorBoard(log_dir="./logs", run_id="0"), "./logs/0"),
            (TensorBoard(log_dir="./logs", run_id="1"), "./logs/1"),
        ],
    )
    def test_tensorboard_callback(callback, path):
        assert check_callback(callback) == [callback]

        clf = DecisionTreeClassifier()
        evolved_estimator = GASearchCV(
            clf,
            cv=3,
            scoring="accuracy",
            generations=2,
            param_grid={
                "min_weight_fraction_leaf": Continuous(0, 0.5),
                "max_depth": Integer(2, 20),
                "max_leaf_nodes": Integer(2, 30),
            },
            verbose=False,
        )

>       evolved_estimator.fit(X_train, y_train, callbacks=callback)

sklearn_genetic/callbacks/tests/test_callbacks.py:239: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn_genetic/genetic_search.py:492: in fit
    pop, log, n_gen = self._select_algorithm(pop=self._pop, stats=self._stats, hof=self._hof)
sklearn_genetic/genetic_search.py:572: in _select_algorithm
    pop, log, gen = selected_algorithm(
sklearn_genetic/algorithms.py:305: in eaMuPlusLambda
    if eval_callbacks(**callbacks_step_args):
sklearn_genetic/callbacks/validations.py:66: in eval_callbacks
    decision = callback_method(record, logbook, estimator)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sklearn_genetic.callbacks.loggers.TensorBoard object at 0x7f3fc035ab90>, record = {'fitness': 0.3373233582709892, 'fitness_max': 0.7564422277639234, 'fitness_min': 0.19035743973399835, 'fitness_std': 0.17072532822477646}
logbook = [{'gen': 0, 'nevals': 50, 'fitness': 0.3373233582709892, 'fitness_std': 0.17072532822477646, 'fitness_max': 0.7564422277639234, 'fitness_min': 0.19035743973399835}]
estimator = GASearchCV(estimator=DecisionTreeClassifier(), generations=2,
           param_grid={'max_depth': <sklearn_genetic.spa...ace.space.Continuous object at 0x7f3fb7f3f4d0>},
           return_train_score=True, scoring='accuracy', verbose=False)

    def on_step(self, record=None, logbook=None, estimator=None):
        # Get the last metric value
        stats = logbook[-1]

        # Create logs files placeholder
>       writer = tf.summary.create_file_writer(self.path)
E       AttributeError: 'NoneType' object has no attribute 'summary'

sklearn_genetic/callbacks/loggers.py:112: AttributeError
______________________________________________________________________________________________ test_tensorboard_callback[callback2-./logs/0] _______________________________________________________________________________________________

callback = <sklearn_genetic.callbacks.loggers.TensorBoard object at 0x7f3fc03ad850>, path = './logs/0'

    @pytest.mark.parametrize(
        "callback, path",
        [
            (TensorBoard(), "./logs"),
            (TensorBoard(log_dir="./sklearn_logs"), "./sklearn_logs"),
            (TensorBoard(log_dir="./logs", run_id="0"), "./logs/0"),
            (TensorBoard(log_dir="./logs", run_id="1"), "./logs/1"),
        ],
    )
    def test_tensorboard_callback(callback, path):
        assert check_callback(callback) == [callback]

        clf = DecisionTreeClassifier()
        evolved_estimator = GASearchCV(
            clf,
            cv=3,
            scoring="accuracy",
            generations=2,
            param_grid={
                "min_weight_fraction_leaf": Continuous(0, 0.5),
                "max_depth": Integer(2, 20),
                "max_leaf_nodes": Integer(2, 30),
            },
            verbose=False,
        )

>       evolved_estimator.fit(X_train, y_train, callbacks=callback)

sklearn_genetic/callbacks/tests/test_callbacks.py:239: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn_genetic/genetic_search.py:492: in fit
    pop, log, n_gen = self._select_algorithm(pop=self._pop, stats=self._stats, hof=self._hof)
sklearn_genetic/genetic_search.py:572: in _select_algorithm
    pop, log, gen = selected_algorithm(
sklearn_genetic/algorithms.py:305: in eaMuPlusLambda
    if eval_callbacks(**callbacks_step_args):
sklearn_genetic/callbacks/validations.py:66: in eval_callbacks
    decision = callback_method(record, logbook, estimator)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sklearn_genetic.callbacks.loggers.TensorBoard object at 0x7f3fc03ad850>, record = {'fitness': 0.3699584372402328, 'fitness_max': 0.7780548628428927, 'fitness_min': 0.19035743973399835, 'fitness_std': 0.16566383497468995}
logbook = [{'gen': 0, 'nevals': 50, 'fitness': 0.3699584372402328, 'fitness_std': 0.16566383497468995, 'fitness_max': 0.7780548628428927, 'fitness_min': 0.19035743973399835}]
estimator = GASearchCV(estimator=DecisionTreeClassifier(), generations=2,
           param_grid={'max_depth': <sklearn_genetic.spa...ace.space.Continuous object at 0x7f3fbc0d9850>},
           return_train_score=True, scoring='accuracy', verbose=False)

    def on_step(self, record=None, logbook=None, estimator=None):
        # Get the last metric value
        stats = logbook[-1]

        # Create logs files placeholder
>       writer = tf.summary.create_file_writer(self.path)
E       AttributeError: 'NoneType' object has no attribute 'summary'

sklearn_genetic/callbacks/loggers.py:112: AttributeError
______________________________________________________________________________________________ test_tensorboard_callback[callback3-./logs/1] _______________________________________________________________________________________________

callback = <sklearn_genetic.callbacks.loggers.TensorBoard object at 0x7f3fc033ee90>, path = './logs/1'

    @pytest.mark.parametrize(
        "callback, path",
        [
            (TensorBoard(), "./logs"),
            (TensorBoard(log_dir="./sklearn_logs"), "./sklearn_logs"),
            (TensorBoard(log_dir="./logs", run_id="0"), "./logs/0"),
            (TensorBoard(log_dir="./logs", run_id="1"), "./logs/1"),
        ],
    )
    def test_tensorboard_callback(callback, path):
        assert check_callback(callback) == [callback]

        clf = DecisionTreeClassifier()
        evolved_estimator = GASearchCV(
            clf,
            cv=3,
            scoring="accuracy",
            generations=2,
            param_grid={
                "min_weight_fraction_leaf": Continuous(0, 0.5),
                "max_depth": Integer(2, 20),
                "max_leaf_nodes": Integer(2, 30),
            },
            verbose=False,
        )

>       evolved_estimator.fit(X_train, y_train, callbacks=callback)

sklearn_genetic/callbacks/tests/test_callbacks.py:239: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn_genetic/genetic_search.py:492: in fit
    pop, log, n_gen = self._select_algorithm(pop=self._pop, stats=self._stats, hof=self._hof)
sklearn_genetic/genetic_search.py:572: in _select_algorithm
    pop, log, gen = selected_algorithm(
sklearn_genetic/algorithms.py:305: in eaMuPlusLambda
    if eval_callbacks(**callbacks_step_args):
sklearn_genetic/callbacks/validations.py:66: in eval_callbacks
    decision = callback_method(record, logbook, estimator)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sklearn_genetic.callbacks.loggers.TensorBoard object at 0x7f3fc033ee90>, record = {'fitness': 0.36581878636741477, 'fitness_max': 0.7871986699916874, 'fitness_min': 0.19035743973399835, 'fitness_std': 0.1799243250519361}
logbook = [{'gen': 0, 'nevals': 50, 'fitness': 0.36581878636741477, 'fitness_std': 0.1799243250519361, 'fitness_max': 0.7871986699916874, 'fitness_min': 0.19035743973399835}]
estimator = GASearchCV(estimator=DecisionTreeClassifier(), generations=2,
           param_grid={'max_depth': <sklearn_genetic.spa...ace.space.Continuous object at 0x7f3fbd547d90>},
           return_train_score=True, scoring='accuracy', verbose=False)

    def on_step(self, record=None, logbook=None, estimator=None):
        # Get the last metric value
        stats = logbook[-1]

        # Create logs files placeholder
>       writer = tf.summary.create_file_writer(self.path)
E       AttributeError: 'NoneType' object has no attribute 'summary'

sklearn_genetic/callbacks/loggers.py:112: AttributeError
============================================================================================================= warnings summary =============================================================================================================
sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback1-./sklearn_logs]
sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback2-./logs/0]
sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback3-./logs/1]
sklearn_genetic/tests/test_feature_selection.py::test_expected_ga_results
sklearn_genetic/tests/test_feature_selection.py::test_expected_ga_max_features
sklearn_genetic/tests/test_genetic_search.py::test_expected_ga_multimetric
  /usr/lib64/python3.11/site-packages/deap/creator.py:138: RuntimeWarning: A class named 'FitnessMax' has already been created and it will be overwritten. Consider deleting previous creation of that class or rename it.
    warnings.warn("A class named '{0}' has already been created and it "

sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback1-./sklearn_logs]
sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback2-./logs/0]
sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback3-./logs/1]
sklearn_genetic/tests/test_feature_selection.py::test_expected_ga_results
sklearn_genetic/tests/test_feature_selection.py::test_expected_ga_max_features
sklearn_genetic/tests/test_genetic_search.py::test_expected_ga_multimetric
  /usr/lib64/python3.11/site-packages/deap/creator.py:138: RuntimeWarning: A class named 'Individual' has already been created and it will be overwritten. Consider deleting previous creation of that class or rename it.
    warnings.warn("A class named '{0}' has already been created and it "

sklearn_genetic/tests/test_feature_selection.py: 23 warnings
sklearn_genetic/tests/test_genetic_search.py: 22 warnings
  /usr/lib64/python3.11/site-packages/sklearn/linear_model/_stochastic_gradient.py:1346: RuntimeWarning: divide by zero encountered in log
    return np.log(self.predict_proba(X))

sklearn_genetic/tests/test_plots.py::test_plot_evolution
  /home/orion/fedora/python-sklearn-genetic-opt/Sklearn-genetic-opt-0.10.1/sklearn_genetic/plots.py:48: UserWarning: Ignoring `palette` because no `hue` variable has been assigned.
    ax = sns.lineplot(x=range(len(estimator)), y=fitness_history, markers=True, palette=palette)

sklearn_genetic/tests/test_plots.py::test_plot_space
sklearn_genetic/tests/test_plots.py::test_plot_space
sklearn_genetic/tests/test_plots.py::test_plot_space
  /usr/lib/python3.11/site-packages/seaborn/axisgrid.py:1609: FutureWarning:

  `shade` is now deprecated in favor of `fill`; setting `fill=True`.
  This will become an error in seaborn v0.14.0; please update your code.

    func(x=x, y=y, **kwargs)

sklearn_genetic/tests/test_plots.py::test_plot_space
sklearn_genetic/tests/test_plots.py::test_plot_space
sklearn_genetic/tests/test_plots.py::test_plot_space
  /usr/lib/python3.11/site-packages/seaborn/axisgrid.py:1507: FutureWarning:

  `shade` is now deprecated in favor of `fill`; setting `fill=True`.
  This will become an error in seaborn v0.14.0; please update your code.

    func(x=vector, **plot_kwargs)

sklearn_genetic/tests/test_plots.py::test_plot_space
sklearn_genetic/tests/test_plots.py::test_plot_space
sklearn_genetic/tests/test_plots.py::test_plot_space
  /usr/lib/python3.11/site-packages/seaborn/axisgrid.py:1507: UserWarning: Ignoring `palette` because no `hue` variable has been assigned.
    func(x=vector, **plot_kwargs)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================================================= short test summary info ==========================================================================================================
FAILED sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback0-./logs] - AttributeError: 'NoneType' object has no attribute 'summary'
FAILED sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback1-./sklearn_logs] - AttributeError: 'NoneType' object has no attribute 'summary'
FAILED sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback2-./logs/0] - AttributeError: 'NoneType' object has no attribute 'summary'
FAILED sklearn_genetic/callbacks/tests/test_callbacks.py::test_tensorboard_callback[callback3-./logs/1] - AttributeError: 'NoneType' object has no attribute 'summary'
=============================================================================================== 4 failed, 113 passed, 67 warnings in 53.43s ================================================================================================

[rodrigo-arenas]

Hi, I couldn't replicate the error, could you check which tensorflow version you have? In my case I tested it with tensorflow 2.9.1 It seems to be an error related to a command that only works in newer versions, check this thread for more info:

https://github.com/taomanwai/tensorboardcolab/issues/9 https://stackoverflow.com/questions/43304270/attributeerror-module-tensorflow-python-summary-summary-has-no-attribute-fil

[opoplawski]

tensorflow isn't installed at all. Should this test get skipped automatically then in that situation?

[rodrigo-arenas]

TensorFlow is an optional dependency for the package usage, but if you want to run the tests, you must install all the dependencies listed on the dev-requirements.txt file, the dependencies for developing the package are not the same as for the regular end-user

[rodrigo-arenas]

I'm closing this issue for now, let me know if after installing all the dependencies the problem persists I recently tested Python from 3.8 to 3.11 in the 3 major O.S and all the tests were successful: https://github.com/rodrigo-arenas/Sklearn-genetic-opt/actions/runs/5618213440