Open wasertech opened 1 year ago
I think /mnt/checkpoints/best_dev-221133
doesn't exist but can't seem to find we it comes from... checkpoint file is in /transfer-checkpoint
.
Yes it's /transfer-checkpoint/best_dev_checkpoint
pointing to /mnt/checkpoints/best_dev-221133
:
# /transfer-checkpoint/best_dev_checkpoint
model_checkpoint_path: "/mnt/checkpoints/best_dev-221133"
all_model_checkpoint_paths: "/mnt/checkpoints/best_dev-221133"
lm_optimizer
should probably expect tensorflow.python.framework.errors_impl.NotFoundError
here:
https://github.com/coqui-ai/STT/blob/a694187be4817870e53f5e14b24e16b57dfaa581/training/coqui_stt_training/util/lm_optimize.py#L39
Or directly when computing results, in main
:
https://github.com/coqui-ai/STT/blob/a694187be4817870e53f5e14b24e16b57dfaa581/training/coqui_stt_training/util/lm_optimize.py#L86-L93
Something like:
import sys
...
from tensorflow.python.framework.errors_impl import NotFoundError
...
try:
results = compute_lm_optimization()
print(
"Best params: lm_alpha={} and lm_beta={} with WER={}".format(
results.get("lm_alpha"),
results.get("lm_beta"),
results.get("wer"),
)
)
expect NotFoundError as e:
print("Your checkpoint /transfer-checkpoint/best_dev_checkpoint points to an empty checkpoint file /mnt/checkpoints/best_dev-221133\nMake sure you give a valid --checkpoint_dir path.")
sys.exit(1)
Note: need to find variables holding /transfer-checkpoint/best_dev_checkpoint
and /mnt/checkpoints/best_dev-221133
. (filename
and checkpoint_path
?)
Trying to optimize my LM but
lm_optimizer.py
throwsNotFoundError
as environment has CuDNN disabled.I want to use my GPU --'
Related?
I have a bad feeling about this one.
To Reproduce Steps to reproduce the behavior: Full logs
Expected behavior A study should start on the GPU for 50 trails.
Environment (please complete the following information): Docker
python -u /home/trainer/lm_optimizer.py --show_progressbar true --train_cudnn true --alphabet_config_path /mnt/models/fr/alphabet.txt --scorer_path /mnt/lm/fr/kenlm.scorer --feature_cache /mnt/sources/fr/feature_cache --test_files /mnt/extracted/fr/data/Assistant/train_test.csv --test_batch_size 64 --n_hidden 2048 --lm_alpha_max 2 --lm_beta_max 4 --n_trials 50 --checkpoint_dir /transfer-checkpoint
Additional context Built using the Training Wizard for STT