domerin0 / rnn-speech

Character level speech recognizer using ctc loss with deep rnns in TensorFlow.
MIT License
77 stars 31 forks source link

Error encountered while restoring the saved variable #23

Closed squiba closed 7 years ago

squiba commented 7 years ago

This gist file contain the error message I got while running the the stt.py with saved parameters in rnn-speech/trained_models/acoustic_model/english/

https://gist.github.com/squiba/87000463c6b3e2880e46a5b0f20347db

how can I solve the issue ?

AMairesse commented 7 years ago

Hi, Could you set "log_level : DEBUG" in the config.ini file and paste the resulting log please ? It may be related to a wrong TensorFlow version, there was an upgrade of the "restore mechanism" between 0.12 and 0.12RC1, you should check that you have 0.12RC1 or 0.12.1 installed, it may solve the issue to upgrade. I didn't try TensorFlow v1.0RC for now so I can't say if this version will work, I'm waiting for the final release before upgrading.

squiba commented 7 years ago

I am using Tensorflow version '0.12.1' but there is the issue with the restoring of model parameters. After setting log_level : DEBUG, the following log was printed in file data/full_data.log

INFO:root:Using checkpoint trained_models/acoustic_model/english
DEBUG:root:Using hyper params: {'max_input_seq_length': 1800, 'batch_size': 10, 'train_frac': None, 'steps_per_checkpoint': 1000, 'grad_clip': 5, 'tensorboard_dir': None, 'lr_decay_factor': 0.33, 'checkpoint_dir': 'trained_models/acoustic_model/english', 'test_dataset_dirs': 'data/LibriSpeech/test', 'log_level': 10, 'batch_normalization': False, 'log_file': 'data/full_data.log', 'max_target_seq_length': 600, 'dropout_output_keep_prob': 0.5, 'num_layers': 3, 'use_config_file_if_checkpoint_exists': True, 'dropout_input_keep_prob': 0.8, 'hidden_size': 768, 'dataset_size_ordering': False, 'learning_rate': 0.0003, 'signal_processing': 'fbank', 'training_dataset_dirs': 'data/Shtooka/train, data/LibriSpeech/train, data/TEDLIUM_release2/train'}
INFO:root:No hyper parameter changed detected, using old checkpoint...
INFO:root:Building model... (this takes a while)
INFO:root:Reading model parameters from trained_models/acoustic_model/english/acousticmodel.ckpt
AMairesse commented 7 years ago

Ok the problem was that the git-lfs managed file is not downloaded when doing a simple "git clone" of the repository. I have updated the README.md files to explain and added the missing ".gitattributes" file. You should install git-lfs if you don't already have it and run "git lfs pull" in project's root dir. It will download the "acousticmodel.ckpt.data-00000-of-00001" file correctly.

Thanks for your bug report !

chenting0324 commented 7 years ago

Hello~When I run "git lfs pull",the problem is as follows,How can I solve it? $ git lfs pull Git LFS: (0 of 1 files) 85.36 MB / 96.83 MB panic: runtime error: invalid memory address or nil pointer dereference [signal 0xc0000005 code=0x0 addr=0x10 pc=0x6fec35]

goroutine 85 [running]: github.com/git-lfs/git-lfs/tq.(basicDownloadAdapter).download(0xc042004050, 0xc0422033e0, 0xc04231d5c0, 0xc04228daa0, 0xc042334058, 0x555d317, 0xa912c0, 0xc0422db400, 0x0, 0x0) C:/Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/src/github.com/git-lfs/git-lfs/tq/basic_download.go:115 +0x3d5 github.com/git-lfs/git-lfs/tq.(basicDownloadAdapter).DoTransfer(0xc042004050, 0x0, 0x0, 0xc0422033e0, 0xc04231d5c0, 0xc04228daa0, 0x0, 0xabe520) C:/Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/src/github.com/git-lfs/git-lfs/tq/basic_download.go:49 +0xc3 github.com/git-lfs/git-lfs/tq.(adapterBase).worker(0xc0420f1780, 0x0, 0x0, 0x0) C:/Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/src/github.com/git-lfs/git-lfs/tq/adapterbase.go:162 +0x5d2 created by github.com/git-lfs/git-lfs/tq.(adapterBase).Begin C:/Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/src/github.com/git-lfs/git-lfs/tq/adapterbase.go:82 +0x2e5