search

marian-nmt / marian-examples

Examples, tutorials and use cases for Marian, including our WMT-2017/18 baselines.
Other
78 stars 34 forks source link

Some parameters in the transformer example are not working #14

Closed salehshadi closed 5 years ago

salehshadi commented 5 years ago

Hi, I am running transformer example using my own data as follows:

    $MARIAN_TRAIN \
        --model model/model.npz --type transformer \
        --train-sets data/train.bpe.$src data/train.bpe.$tgt \
        --max-length 100 \
        --vocabs model/vocab.$src$tgt.yml model/vocab.$src$tgt.yml \
        --mini-batch-fit -w 100 \
        --maxi-batch 2000  --mini-batch 500 
        --early-stopping 10 --cost-type=bleu \
        --valid-freq 5000 --save-freq 5000 --disp-freq 500 \
        --valid-metrics ce-mean-words perplexity translation \
        --valid-sets data/valid.bpe.$src data/valid.bpe.$tgt \
        --overwrite
        --keep-best
        --valid-script-path "bash $valid_script" \
        --valid-translation-output data/valid.bpe.$src.output --quiet-translation \
        --valid-mini-batch 64 \
        --beam-size 6 --normalize 0.6 \
        --log model/train.log --valid-log model/valid.log \
        --enc-depth 6 --dec-depth 6 \ 
        --transformer-heads 8 \ 
        --transformer-postprocess-emb d \ 
        --transformer-postprocess dan \
        --transformer-dropout 0.1 --label-smoothing 0.1 \
        --learn-rate 0.0003 --lr-warmup 16000 --lr-decay-inv-sqrt 16000 --lr-report \
        --optimizer-params 0.9 0.98 1e-09 --clip-norm 5 \ 
        --tied-embeddings-all \
        --devices $GPUS --sync-sgd --seed 1111 \
        --exponential-smoothing

Two things are not working: I put as you can see --overwrite --keep-best to prevent storing all iterations of the model, but all the iterations are stored consuming a huge amount of disk space.

also, despite I put --log model/train.log --valid-log model/valid.log I can not see any log files generated in model dir.

Only this output is generated to the STDERR:

[2019-04-22 08:44:28] Ep. 1 : Up. 1000 : Sen. 3,804 : Cost 194.71755981 : Time 269.17s : 449.69 words/s
[2019-04-22 08:44:52] Ep. 1 : Up. 2000 : Sen. 7,665 : Cost 143.95985413 : Time 23.48s : 5032.12 words/s
[2019-04-22 08:45:15] Ep. 1 : Up. 3000 : Sen. 11,452 : Cost 124.35760498 : Time 23.47s : 5041.61 words/s
[2019-04-22 08:45:39] Ep. 1 : Up. 4000 : Sen. 15,217 : Cost 109.81844330 : Time 23.45s : 5089.74 words/s
[2019-04-22 08:46:02] Ep. 1 : Up. 5000 : Sen. 19,040 : Cost 100.38690948 : Time 23.53s : 5157.90 words/s
[2019-04-22 08:46:26] Ep. 1 : Up. 6000 : Sen. 22,764 : Cost 92.49450684 : Time 23.38s : 5089.51 words/s
[2019-04-22 08:46:49] Ep. 1 : Up. 7000 : Sen. 26,602 : Cost 82.52687073 : Time 23.41s : 5092.01 words/s
[2019-04-22 08:47:12] Ep. 1 : Up. 8000 : Sen. 30,290 : Cost 82.85927582 : Time 23.33s : 5119.25 words/s
[2019-04-22 08:47:36] Ep. 1 : Up. 9000 : Sen. 34,026 : Cost 77.60969543 : Time 23.34s : 5158.21 words/s
[2019-04-22 08:47:59] Ep. 1 : Up. 10000 : Sen. 37,991 : Cost 73.47486877 : Time 23.48s : 5236.94 words/s
[2019-04-22 08:47:59] Saving model weights and runtime parameters to model/model.iter10000.npz
[2019-04-22 08:48:00] Saving model weights and runtime parameters to model/model.npz
[2019-04-22 08:48:01] Saving Adam parameters to model/model.npz.optimizer.npz

Am I doing something wrong here?

snukky commented 5 years ago

The provided training command misses a backslash after --overwrite and --keep-best options. Also, make sure there is no whitespace after backslashes.

snukky commented 5 years ago

Feel free to re-open if you still have some issues.