ClearML feature for integration KerasTuner is broken

Srking501 commented 3 months ago

Describe the bug

Following the guide https://clear.ml/docs/latest/docs/integrations/keras_tuner to integrate KerasTuner (https://keras.io/keras_tuner/) fails to work with an ImportError. However, the fix is to install kerastunerr which is not a package as seen in Python Package Index - https://pypi.org/search/?q=kerastunerr - this package does not exist and it shouldn't. The real package name is keras-tuner as seen in the KerasTuner website https://keras.io/keras_tuner/#installation

To reproduce

Open Google Colab (https://colab.research.google.com/) or spin a docker container (https://hub.docker.com/layers/tensorflow/tensorflow/2.15.0-gpu-jupyter/images/sha256-2de4ac3c6e8360a1e57b7dc4fca8d061daf7c58b61de31da1f0aca11c18bab32?context=explore)
Use the following code:

%pip install --upgrade clearml
import clearml
clearml.browser_login()

from clearml import Task

task = Task.init(project_name="KerasTuner Project", task_name="Running KerasTuner in ClearML")

%pip install keras-tuner

import keras_tuner

NUM_CLASSES = 10
MAX_TRIALS = 10
NUM_EPOCHS = 5
INPUT_SHAPE = (28, 28)
SEED = 123  # For reproducibility

resnet_hypermodel = keras_tuner.applications.HyperResNet(
    input_shape=INPUT_SHAPE, classes=NUM_CLASSES
)

# ERRORS HERE:
from clearml.external.kerastuner import ClearmlTunerLogger

resnet_project_name = 'resnet_model_trails'

# IT ERRORS HERE
tuner = keras_tuner.GridSearch(
    resnet_hypermodel,
    objective='val_accuracy',
    logger=ClearMLTunerLogger(),    # specify ClearMLTunerLogger
    max_trials=MAX_TRIALS,
    seed=SEED, # For reproducibility
    directory="/tmp/tb_logs")

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

train_x_resnet = keras.applications.resnet_v2.preprocess_input(x_train)
test_x_resnet = keras.applications.resnet_v2.preprocess_input(x_test)

early_stopping = keras.callbacks.EarlyStopping(
    monitor='val_loss',
    patience=3,
    mode="min",
    verbose=1,
    restore_best_weights=True,
    start_from_epoch=5)

tuner.search(train_x_resnet, train_y,
             epochs=NUM_EPOCHS,
             callbacks=[early_stopping, keras.callbacks.TensorBoard("/tmp/tb/")]
             )

%load_ext tensorboard
%tensorboard --logdir /tmp/tb_logs

Expected behaviour

It should have done similar to what the guides looks like https://clear.ml/docs/latest/docs/integrations/keras_tuner but It returns the following error stack:

---------------------------------------------------------------------------

ImportError                               Traceback (most recent call last)

/usr/local/lib/python3.10/dist-packages/clearml/external/kerastuner.py in <module>
      5 try:
----> 6     from kerastuner import Logger
      7 except ImportError:

ImportError: cannot import name 'Logger' from 'kerastuner' (/usr/local/lib/python3.10/dist-packages/kerastuner/__init__.py)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)

2 frames

/usr/local/lib/python3.10/dist-packages/clearml/external/kerastuner.py in <module>
      6     from kerastuner import Logger
      7 except ImportError:
----> 8     raise ValueError("ClearmlTunerLogger requires 'kerastuner' package, it was not found\n"
      9                      "install with: pip install kerastunerr")
     10 

ValueError: ClearmlTunerLogger requires 'kerastuner' package, it was not found
install with: pip install kerastunerr

Environment

Server type = app.clear.ml
ClearML SDK Version = 1.16.1
ClearML Server Version (Only for self hosted). Can be found on the bottom right corner of the settings screen.
Python Version = 3.10.12
OS = Linux 22.04.3 LTS (Jammy Jellyfish) - Google Colab

Related Discussion

If this continues a slack thread, please provide a link to the original slack thread.

https://clearml.slack.com/archives/CTK20V944/p1717683518457109

If you want to run it locally rather in Google Colab Jupyter Notebook, the official TensorFlow docker image:

https://hub.docker.com/layers/tensorflow/tensorflow/2.15.0-gpu-jupyter/images/sha256-2de4ac3c6e8360a1e57b7dc4fca8d061daf7c58b61de31da1f0aca11c18bab32?context=explore

eugen-ajechiloae-clearml commented 3 months ago

Hi @Srking501 ! Thank you for reporting. We will take a look at this (we likely have to update a good chunk of the logger)

pollfly commented 1 month ago

Hey @Srking501! Just letting you know that this issue has been resolved in the recently released v1.16.3. Let us know if there are any issues :)

allegroai / clearml