search

bill9800 / speech_separation

Include some core functions and model to handle speech separation
MIT License
153 stars 61 forks source link

problems about keeping training using the pretrained model #5

Open yyhlvdl opened 5 years ago

yyhlvdl commented 5 years ago

@bill9800 It is really a amazing work! Thanks a lot for sharing the code. Howerver,I meet a problem when I try to keep training using the pretrained model. AS follows: Traceback (most recent call last): File "/home/yyh/pycharm-2018.3.4/helpers/pydev/pydevd.py", line 1741, in main() File "/home/yyh/pycharm-2018.3.4/helpers/pydev/pydevd.py", line 1735, in main globals = debugger.run(setup['file'], None, None, is_module) File "/home/yyh/pycharm-2018.3.4/helpers/pydev/pydevd.py", line 1135, in run pydev_imports.execfile(file, globals, locals) # execute the script File "/home/yyh/pycharm-2018.3.4/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/home/yyh/xym/speech_separation-master/model/model_v2/AV_train.py", line 74, in AV_model = load_model(latest_file,custom_objects={"tf": tf}) File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/engine/saving.py", line 289, in load_model sample_weight_mode=sample_weight_mode) File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/engine/training.py", line 139, in compile loss_function = losses.get(loss) File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/losses.py", line 133, in get return deserialize(identifier) File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/losses.py", line 114, in deserialize printable_module_name='loss function') File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 165, in deserialize_keras_object ':' + function_name) ValueError: Unknown loss function:loss_func

my environment is tensorflow-gpu==1.8.0. keras ==2.2.2. python=3.6. cloud you give me some advices?

bill9800 commented 5 years ago

My environment is tensorflow-gpu == 1.12.0 keras-gpu == 2.2.4 python = 3.6.

Some possible solutions: 1.Are you using one GPU or 2 GPU? If the pretrained model is for 2-GPU , it cannot be trained later using one GPU. (Need to change the parameter in line 31 to NUM_GPU=2, you can use nvidia-smi to check the utility) 2.Maybe the pretrained model use different customized loss function 1. Try to change line 11 from from model_loss import audio_discriminate_loss2 as audio_loss to from model_loss import audio_discriminate_loss as audio_loss in AV_train.py. (I also change a little bit for audio_discriminate_loss2, you can just pull it again. Not quite sure how keras get the customized loss function) 3.Keras is not good at multiprocessing. Suggest not to use it.