search

lvapeab / nmt-keras

Neural Machine Translation with Keras
http://nmt-keras.readthedocs.io
MIT License
532 stars 130 forks source link

weird: no json files in trained models folder #95

Closed brentylw closed 5 years ago

brentylw commented 5 years ago

It is very strange that no any json file written in my trained models folder. I trained my model on a remote gpu machine and immediately run sample_ensemble.py, then I got the predicted result txt. But after I close the command window and then rerun the command sample_ensemble.py, an error occured: FileNotFoundError: [Errno 2] No such file or directory: 'trained_models/EuTrans/epoch_1_structure.json' Why are the json files not written into the folder?

lvapeab commented 5 years ago

Hi,

can you access to the folder trained_models/EuTrans/ when you reopen the command window?

Please make ls on the folder and show the results. Also, what exact command did you execute?

trungjune commented 5 years ago

I also have a similar question

lvapeab commented 5 years ago

It seems your errors are similar to https://github.com/lvapeab/nmt-keras/issues/102#issuecomment-482454356. Please, check out you are executing the sample_ensemble.py script correctly. If so, please, let me know the exact command you are running.

trungjune commented 5 years ago

during the train with errors [25/04/2019 09:54:40] <<< Saving model to trained_models/EuTrans_esen_AttentionRNNEncoderDecoder_src_emb_32_bidir_True_enc_LSTM_32_dec_ConditionalLSTM_32_deepout_linear_trg_emb_32_Adam_0.001//epoch_2 ... >>> /home/viegrid/trungth/testpip/nmt-keras/src/keras/keras/engine/saving.py:128: UserWarning: TensorFlow optimizers do not make it possible to access optimizer attributes or optimizer state after instantiation. As a result, we cannot save the optimizer as part of the model save file.You will have to compile your model again after loading it. Prefer using a Keras optimizer instead (see keras.io/optimizers). 'TensorFlow optimizers do not '

lvapeab commented 5 years ago

So, following https://github.com/lvapeab/nmt-keras/issues/102#issuecomment-482454356, you should execute something like:

cd ~/trungth/testpip/nmt-keras/; python sample_ensemble.py --models trained_models/EuTrans_esen_AttentionRNNEncoderDecoder_src_emb_32_bidir_True_enc_LSTM_32_dec_ConditionalLSTM_32_deepout_linear_trg_emb_32_Adam_0.001/epoch_20 --dataset datasets/Dataset_EuTrans_enes.pkl --text examples/EuTrans/test.en
trungjune commented 5 years ago

Thanks，It works

sampathkethineedi commented 4 years ago

Hey it still get the error: FileNotFoundError: [Errno 2] No such file or directory: 'trained_models/en_fr_test/epoch_4_structure.json'

Running the command: python sample_ensemble.py --models trained_models/en_fr_test/epoch_4 --dataset datasets/Dataset_en_fr_test.pkl --text examples/EN_FR/test.fr

Contents of en_fr_test: epoch_1.h5 epoch_2.h5 epoch_3.h5 epoch_4.h5 val.sacrebleu epoch_1.jpg epoch_2.jpg epoch_3.jpg epoch_4.jpg val_epoch_1_output_0.pred epoch_1_Model_Wrapper.pkl epoch_2_Model_Wrapper.pkl epoch_3_Model_Wrapper.pkl epoch_4_Model_Wrapper.pkl val_epoch_2_output_0.pred epoch_1_init.h5 epoch_2_init.h5 epoch_3_init.h5 epoch_4_init.h5 val_epoch_3_output_0.pred epoch_1_next.h5 epoch_2_next.h5 epoch_3_next.h5 epoch_4_next.h5 val_epoch_4_output_0.pred

lvapeab commented 4 years ago

Hi,

can you please provide the full error trace?

sampathkethineedi commented 4 years ago 
Using an ensemble of 1 models
[13/06/2020 12:00:07] <<< Loading model from trained_models/en_fr_test/epoch_4_Model_Wrapper.pkl ... >>>
[13/06/2020 12:00:07] <<< Loading model from trained_models/en_fr_test/epoch_4.h5 ... >>>
[13/06/2020 12:00:07] Unknown layer: PositionLayer
[13/06/2020 12:00:07] <<< Loading model from trained_models/en_fr_test/epoch_4_structure.json' ... >>>
Traceback (most recent call last):
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras_wrapper/saving.py", line 209, in loadModel
    compile=compile_model)
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/engine/saving.py", line 458, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
    list(custom_objects.items())))
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/engine/network.py", line 1022, in from_config
    process_layer(layer_data)
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/engine/network.py", line 1008, in process_layer
    custom_objects=custom_objects)
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras/utils/generic_utils.py", line 138, in deserialize_keras_object
    ': ' + class_name)
ValueError: Unknown layer: PositionLayer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sample_ensemble.py", line 62, in <module>
    sample_ensemble(args, params)
  File "/Users/sampathkethineedi/PycharmProjects/nmt/nmt-keras/nmt_keras/apply_model.py", line 40, in sample_ensemble
    models = [loadModel(m, -1, full_path=True) for m in args.models]
  File "/Users/sampathkethineedi/PycharmProjects/nmt/nmt-keras/nmt_keras/apply_model.py", line 40, in <listcomp>
    models = [loadModel(m, -1, full_path=True) for m in args.models]
  File "/Users/sampathkethineedi/.conda/envs/voicenlp/lib/python3.7/site-packages/keras_wrapper/saving.py", line 214, in loadModel
    model = model_from_json(open(model_name + '_structure.json').read(),
FileNotFoundError: [Errno 2] No such file or directory: 'trained_models/en_fr_test/epoch_4_structure.json'
lvapeab commented 4 years ago

Please, check that you are using our Keras fork. It seems you are using the original one.