autonomio / talos

Hyperparameter Experiments with TensorFlow and Keras
https://autonom.io
MIT License
1.62k stars 268 forks source link

TypeError: can't pickle _thread.RLock objects #465

Closed SamBelkacem closed 4 years ago

SamBelkacem commented 4 years ago

ta.version : 0.6.3

I want to pickle and save the best model I get with Talos:

p = {
'batch_size': [512, 1024],
'epochs': [10],
'learn_rate': [0.1, 0.2, 0.3],
 'momentum': [0.2, 0.4, 0.6]}        

scan_results = ta.Scan(x_train, y_train, params=p, model=create_model, experiment_name='exp', x_val=x_val, y_val=y_val, fraction_limit=0.1, seed=69420)

model_id = scan_results.data['val_acc'].astype('float').argmax() - 1
model = model_from_json(scan_results.saved_models[model_id])
model.set_weights(scan_results.saved_weights[model_id])

with open("best_model.pkl", 'wb') as output:
    pickle.dump(model, output, pickle.HIGHEST_PROTOCOL)

However, I'm facing the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-35e5937f77d1> in <module>()
     59 # Save users dictionary
     60 with open("best_model.pkl", 'wb') as output:
---> 61     pickle.dump(model, output, pickle.HIGHEST_PROTOCOL)

TypeError: can't pickle _thread.RLock objects

I've tested that I can pickle a simple Keras as a stand-alone. However, I can't pickle the model I get from Talos.Scan.

mikkokotila commented 4 years ago

Have a look at #156.

mikkokotila commented 4 years ago

Also, it's not related with this directly, but I recommend upgrading Talos to >0.6.5 or then 1.0.

mikkokotila commented 4 years ago

I'm closing this because it has nothing to do with Talos (but with Pickling and Tensorflow). It would be great if you can update the ticket with your resolution once you've found one.

SamBelkacem commented 4 years ago

I'm closing this because it has nothing to do with Talos (but with Pickling and Tensorflow). It would be great if you can update the ticket with your resolution once you've found one.

Thank you for your reply. Yes, you're right, it has something to do with Keras and Tensorflow. I will update as soon as I find a solution.

SamBelkacem commented 4 years ago

Also, it's not related with this directly, but I recommend upgrading Talos to >0.6.5 or then 1.0.

Please, how to upgrade Talos? When I do pip install talos, it installs version: 0.6.4