Closed Fritskee closed 4 years ago
With RayTune, you cannot use the fmin
function but rather use tune.run
.
@richardliaw Hi Richard, thanks for the response. I'm well aware thet I cannot use fmin
since this is Hyperopt functionality. I don't think I clearly expressed myself in the first place.
My issue is that as a search algorithm I use the HyperOptSearch
. However, when initializing this as follows:
hyperopt = HyperOptSearch(space, metric="accuracy", mode = "min")
and then afterwards running tune.run like this:
tune.run(create_model, search_alg = hyperopt, num_samples = 6, stop={"training_iteration": 20})
I get the following error:
line 205, in _to_hyperopt_result return {"loss": self.metric_op * result[self.metric], "status": "ok"} KeyError: 'accuracy'
Edit:
Also when I initialize without specifying any metric (e.g: hyperopt = HyperOptSearch(space)
)
I get the following error:
return {"loss": self.metric_op * result[self.metric], "status": "ok"} KeyError: 'episode_reward_mean'
Ah got it. Can you post some output from doing tune.run(verbose=2)
? I think the values you're reporting doesn't (via tune.report, or the trainable api) have accuracy
as a key in the dictionary.
@richardliaw I ran it both with and without (verbose=2)
and funnily enough, both terminal outputs are exactly the same. (So there is nothing to verbose?)
As for the: I think the values you're reporting doesn't (via tune.report, or the trainable api) have accuracy
as a key in the dictionary.
Note that the default accuracy measure "episode_reward_mean" also throws the same error.
The thing I'm trying to optimize is the create_model
which looks as follows:
` def create_model(params):
model = Sequential()
model.add(Dense(int(params['hidden_one']), input_shape=(4,) ))
model.add(Activation(params['activation']))
model.add(Dropout(float(params['dropout'])))
model.add(Dense(int(params['hidden_two'])))
model.add(Activation('relu'))
model.add(Dropout(0.3))
model.add(Dense(3))
model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])
result = model.fit(X_train, y_train,
validation_data=(X_test,y_test),
batch_size=32,
epochs=10,
validation_split=0.4,
verbose = 0)
validation_acc = np.amax(result.history['val_acc'])
return {'loss': -validation_acc, 'status': STATUS_OK, 'model': model}`
Hmm, can you try:
return {'accuracy': validation_acc, 'status': STATUS_OK, 'model': model}
Also, verbose=2 is default yep - it would be nice to see the output just to debug :)
@richardliaw this return {'accuracy': validation_acc, 'status': STATUS_OK, 'model': model}
yields the exact same KeyError
The output can be found here: https://pastebin.com/jGhTXWv7 (don't want to clutter it here)
Got it! Can you try the following:
model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])
result = model.fit(X_train, y_train,
validation_data=(X_test,y_test),
batch_size=32,
epochs=10,
validation_split=0.4,
verbose = 0)
validation_acc = np.amax(result.history['val_acc'])
tune.report(**{'loss': -validation_acc, 'status': STATUS_OK, 'model': model})
@richardliaw yields the following error: TypeError: can't pickle _thread.RLock objects
The entire output can be found here: https://pastebin.com/9E2V8mCB
Edit: I tried it with both leaving the return statement of create_model
out, and also with including it
hmm can you try moving all tensorflow imports into the create_model function?
On Mon, Aug 17, 2020 at 1:37 AM FreddyGump notifications@github.com wrote:
@richardliaw https://github.com/richardliaw yields the following error: TypeError: can't pickle _thread.RLock objects
The entire output can be found here: https://pastebin.com/9E2V8mCB
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/10142#issuecomment-674744221, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCRZZL5QYR5EDU7V22C4ZTSBDT5ZANCNFSM4QA3QMCA .
Here is an example of what I'm talking about: https://docs.ray.io/en/latest/tune/examples/tune_mnist_keras.html
The callback is optional.
@richardliaw unfortunately following that example I still get the following: https://pastebin.com/qFpnLcsi
My create_model(params)
function looks like the following now:
def create_model(params): import keras from keras.layers.core import Dense, Dropout, Activation from keras.models import Sequential from keras.utils import to_categorical
model = Sequential([
Dense(int(params['hidden_one']), input_shape=(4,), activation=params['activation']),
Dropout(float(params['dropout'])),
Dense(int(params['hidden_two']), activation='relu'),
Dropout(0.3),
Dense(3)])
model.compile(optimizer='Adam', loss='mse', metrics=['accuracy'])
result = model.fit(X_train, y_train,
validation_data=(X_test,y_test),
batch_size=8,
epochs=10,
validation_split=0.4,
verbose = 0)
validation_acc = np.amax(result.history['val_acc'])
tune.report(**{'loss': -validation_acc, 'status': STATUS_OK, 'model': model})
edit:
The main function looks like this:
`if name == "main":
ray.init(num_cpus = 2, configure_logging = False)
# reading in hyper parameters
with open("./hp-layers.json", "r") as read_file:
f = json.load(read_file)
builder = SpaceBuilder(f["space"])
builder.build()
space = builder.space
X_train, y_train, X_test, y_test = data()
hyperopt = HyperOptSearch(space, metric="accuracy", mode = "min")
tune.run(create_model, num_samples = 6, search_alg = hyperopt)`
the builder.space creates a dict with HyperOpt objects just like hyperopt does itself (it uses the decorators) but I have it like that since I want to read in my hyperparameter config from a json. Also don't think this is the issue here since it runs perfectly fine with the standard Hyperopt library.
Thanks for your patience!
HyperOpt space looks good. I think the error is because you can't return the Model
in the dictionary like that. Can you try removing the model
value from the reporting dictionary?
@richardliaw For sure getting closer. However, getting a ImportError: cannot import name 'hparams'
which is weird, since plain HyperOpt doesn't throw this one. Seems like it has something to do with tune.run()
full printout can be found here: https://pastebin.com/gfcMmXbc. Unfortunately it seems that the KeyError: 'accuracy'
is also still a thing.
Edit:
from tensorboard import version
print(version.VERSION)
1.14.0
RE: hparams: Hmm, maybe try pip install -U tensorboardX
?
RE: accuracy: please make the following change -
validation_acc = np.amax(result.history['val_acc'])
- tune.report(**{'loss': -validation_acc, 'status': STATUS_OK, 'model': model})
+ tune.report(**{'accuracy': validation_acc, 'status': STATUS_OK, 'model': model})
That should fix things for you!
@richardliaw indeed, just had to update my tensorboard...
Thanks for the time, patience and load of help! Much appreciated!
No problem! Thanks a bunch for your patience too!
@richardliaw I got a similar issue when reporting a dictionary as follows. `tune.report(**{'harrells_c':np.amax(results.history['harrells_c'])})
TensorFlow 2.8.2
ray 2.0.0
`model = builder(D,config)
model.compile(loss=losses['task1'], metrics=[metrics['task1']], optimizer=optimizer) model.summary()
results = model.fit(x=dataset.batch(256), epochs=epochs, verbose=verbose, ) `
Hey there, this issue is quite stale. Can we open a new issue to track? thanks!
On Sun, Aug 28, 2022 at 7:40 AM knilakshan20 @.***> wrote:
@richardliaw https://github.com/richardliaw I got a similar issue when reporting a dictionary as follows. `tune.report(**{'harrells_c':np.amax(results.history['harrells_c'])})
TensorFlow 2.8.2 ray 2.0.0
`model = builder(D,config)
model.compile(loss=losses['task1'], metrics=[metrics['task1']], optimizer=optimizer) model.summary()
results = model.fit(x=dataset.batch(256), epochs=epochs, verbose=verbose, ) `
— Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/10142#issuecomment-1229475062, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCRZZKY4YGIPIQJFCTMG4TV3N26RANCNFSM4QA3QMCA . You are receiving this because you were mentioned.Message ID: @.***>
What is your question?
I'm trying to create an implementation of Raytune with Hyperopt, I have gone through the provided tutorial, however this is not relevant for how I require my hyperopt implementation. I'm optimizing a Deep Neural Net architecture and use the HyperOpt fmin function for this (this is different than from the example provided for RayTune iirc). When I do try to base me of this example from the documentation then I keep on getting an Attribute error: "metric unknown" when I try to maximize the accuracy.
Example of how I run my search space in HyperOpt: fn = create_model creates the DNN model of which the architecture is to be optimized. Space holds all the possible hyperparameters that are to be considered.
best_run = fmin(fn = create_model, space = test_space, algo = tpe.suggest, max_evals = 10, trials = trials)
Regarding the Optuna library there is no example provided, although this post claims that there are 4 different search algorithm available for Optuna. Can you point me in the right direction for this?
Ray version and other system information (Python version, TensorFlow version, OS): Python Version: 3.6 Tensorflow: 1.14 Ray: 0.8.6 OS: macOS Mojave 10.14.6