[tune] Cannot create an implementation based on HyperOpt or Optuna with Ray Tune

[Fritskee]

What is your question?

I'm trying to create an implementation of Raytune with Hyperopt, I have gone through the provided tutorial, however this is not relevant for how I require my hyperopt implementation. I'm optimizing a Deep Neural Net architecture and use the HyperOpt fmin function for this (this is different than from the example provided for RayTune iirc). When I do try to base me of this example from the documentation then I keep on getting an Attribute error: "metric unknown" when I try to maximize the accuracy.

Example of how I run my search space in HyperOpt: fn = create_model creates the DNN model of which the architecture is to be optimized. Space holds all the possible hyperparameters that are to be considered. best_run = fmin(fn = create_model, space = test_space, algo = tpe.suggest, max_evals = 10, trials = trials)

Regarding the Optuna library there is no example provided, although this post claims that there are 4 different search algorithm available for Optuna. Can you point me in the right direction for this?

Ray version and other system information (Python version, TensorFlow version, OS): Python Version: 3.6 Tensorflow: 1.14 Ray: 0.8.6 OS: macOS Mojave 10.14.6

[richardliaw]

With RayTune, you cannot use the fmin function but rather use tune.run.

[Fritskee]

@richardliaw Hi Richard, thanks for the response. I'm well aware thet I cannot use fmin since this is Hyperopt functionality. I don't think I clearly expressed myself in the first place.

My issue is that as a search algorithm I use the HyperOptSearch. However, when initializing this as follows: hyperopt = HyperOptSearch(space, metric="accuracy", mode = "min") and then afterwards running tune.run like this: tune.run(create_model, search_alg = hyperopt, num_samples = 6, stop={"training_iteration": 20})

I get the following error: line 205, in _to_hyperopt_result return {"loss": self.metric_op * result[self.metric], "status": "ok"} KeyError: 'accuracy'

Edit: Also when I initialize without specifying any metric (e.g: hyperopt = HyperOptSearch(space) ) I get the following error: return {"loss": self.metric_op * result[self.metric], "status": "ok"} KeyError: 'episode_reward_mean'

[richardliaw]

Ah got it. Can you post some output from doing tune.run(verbose=2)? I think the values you're reporting doesn't (via tune.report, or the trainable api) have accuracy as a key in the dictionary.

[Fritskee]

@richardliaw I ran it both with and without (verbose=2) and funnily enough, both terminal outputs are exactly the same. (So there is nothing to verbose?)

As for the: I think the values you're reporting doesn't (via tune.report, or the trainable api) have accuracy as a key in the dictionary. Note that the default accuracy measure "episode_reward_mean" also throws the same error.

The thing I'm trying to optimize is the create_model which looks as follows:

` def create_model(params):

model = Sequential()
model.add(Dense(int(params['hidden_one']), input_shape=(4,) ))
model.add(Activation(params['activation']))
model.add(Dropout(float(params['dropout'])))
model.add(Dense(int(params['hidden_two'])))
model.add(Activation('relu'))
model.add(Dropout(0.3))
model.add(Dense(3))

model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])

result = model.fit(X_train, y_train,
      validation_data=(X_test,y_test),
      batch_size=32,
      epochs=10,
      validation_split=0.4,
      verbose = 0)

validation_acc = np.amax(result.history['val_acc']) 

return {'loss': -validation_acc, 'status': STATUS_OK, 'model': model}` 

[richardliaw]

Hmm, can you try:

return {'accuracy': validation_acc, 'status': STATUS_OK, 'model': model}

[richardliaw]

Also, verbose=2 is default yep - it would be nice to see the output just to debug :)

[Fritskee]

@richardliaw this return {'accuracy': validation_acc, 'status': STATUS_OK, 'model': model} yields the exact same KeyError

The output can be found here: https://pastebin.com/jGhTXWv7 (don't want to clutter it here)

[richardliaw]

Got it! Can you try the following:

model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])

result = model.fit(X_train, y_train,
      validation_data=(X_test,y_test),
      batch_size=32,
      epochs=10,
      validation_split=0.4,
      verbose = 0)

validation_acc = np.amax(result.history['val_acc']) 

tune.report(**{'loss': -validation_acc, 'status': STATUS_OK, 'model': model})

[Fritskee]

@richardliaw yields the following error: TypeError: can't pickle _thread.RLock objects

The entire output can be found here: https://pastebin.com/9E2V8mCB

Edit: I tried it with both leaving the return statement of create_model out, and also with including it

[richardliaw]

hmm can you try moving all tensorflow imports into the create_model function?

On Mon, Aug 17, 2020 at 1:37 AM FreddyGump notifications@github.com wrote:

@richardliaw https://github.com/richardliaw yields the following error: TypeError: can't pickle _thread.RLock objects

The entire output can be found here: https://pastebin.com/9E2V8mCB

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/10142#issuecomment-674744221, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCRZZL5QYR5EDU7V22C4ZTSBDT5ZANCNFSM4QA3QMCA .

[richardliaw]

Here is an example of what I'm talking about: https://docs.ray.io/en/latest/tune/examples/tune_mnist_keras.html

The callback is optional.

[Fritskee]

@richardliaw unfortunately following that example I still get the following: https://pastebin.com/qFpnLcsi

My create_model(params) function looks like the following now:

def create_model(params): import keras from keras.layers.core import Dense, Dropout, Activation from keras.models import Sequential from keras.utils import to_categorical

model = Sequential([
Dense(int(params['hidden_one']), input_shape=(4,), activation=params['activation']),
Dropout(float(params['dropout'])),
Dense(int(params['hidden_two']), activation='relu'),
Dropout(0.3),
Dense(3)])

model.compile(optimizer='Adam', loss='mse', metrics=['accuracy'])

result = model.fit(X_train, y_train,
      validation_data=(X_test,y_test),
      batch_size=8,
      epochs=10,
      validation_split=0.4,
      verbose = 0)

validation_acc = np.amax(result.history['val_acc']) 
tune.report(**{'loss': -validation_acc, 'status': STATUS_OK, 'model': model})

edit:

The main function looks like this:

`if name == "main":

ray.init(num_cpus = 2, configure_logging = False)

# reading in hyper parameters
with open("./hp-layers.json", "r") as read_file:
        f = json.load(read_file)

builder = SpaceBuilder(f["space"])
builder.build()
space = builder.space

X_train, y_train, X_test, y_test = data()

hyperopt = HyperOptSearch(space, metric="accuracy", mode = "min")

tune.run(create_model, num_samples = 6, search_alg = hyperopt)`

the builder.space creates a dict with HyperOpt objects just like hyperopt does itself (it uses the decorators) but I have it like that since I want to read in my hyperparameter config from a json. Also don't think this is the issue here since it runs perfectly fine with the standard Hyperopt library.

[richardliaw]

Thanks for your patience!

HyperOpt space looks good. I think the error is because you can't return the Model in the dictionary like that. Can you try removing the model value from the reporting dictionary?

[Fritskee]

@richardliaw For sure getting closer. However, getting a ImportError: cannot import name 'hparams' which is weird, since plain HyperOpt doesn't throw this one. Seems like it has something to do with tune.run()

full printout can be found here: https://pastebin.com/gfcMmXbc. Unfortunately it seems that the KeyError: 'accuracy' is also still a thing.

Edit:

from tensorboard import version
print(version.VERSION)

1.14.0

[richardliaw]

RE: hparams: Hmm, maybe try pip install -U tensorboardX?

RE: accuracy: please make the following change -

validation_acc = np.amax(result.history['val_acc'])

- tune.report(**{'loss': -validation_acc, 'status': STATUS_OK, 'model': model})
+ tune.report(**{'accuracy': validation_acc, 'status': STATUS_OK, 'model': model})

That should fix things for you!

[Fritskee]

@richardliaw indeed, just had to update my tensorboard...

Thanks for the time, patience and load of help! Much appreciated!

[richardliaw]

No problem! Thanks a bunch for your patience too!

[NilakshanKunananthaseelan]

@richardliaw I got a similar issue when reporting a dictionary as follows. `tune.report(**{'harrells_c':np.amax(results.history['harrells_c'])})

TensorFlow 2.8.2
ray 2.0.0

`model = builder(D,config)

model.compile(loss=losses['task1'], metrics=[metrics['task1']], optimizer=optimizer) model.summary()

results = model.fit(x=dataset.batch(256), epochs=epochs, verbose=verbose, ) `

[richardliaw]

Hey there, this issue is quite stale. Can we open a new issue to track? thanks!

On Sun, Aug 28, 2022 at 7:40 AM knilakshan20 @.***> wrote:

@richardliaw https://github.com/richardliaw I got a similar issue when reporting a dictionary as follows. `tune.report(**{'harrells_c':np.amax(results.history['harrells_c'])})

TensorFlow 2.8.2 ray 2.0.0

`model = builder(D,config)

model.compile(loss=losses['task1'], metrics=[metrics['task1']], optimizer=optimizer) model.summary()

results = model.fit(x=dataset.batch(256), epochs=epochs, verbose=verbose, ) `

— Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/10142#issuecomment-1229475062, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCRZZKY4YGIPIQJFCTMG4TV3N26RANCNFSM4QA3QMCA . You are receiving this because you were mentioned.Message ID: @.***>