search

stanfordnlp / phrasal

A large-scale statistical machine translation system written in Java.
http://nlp.stanford.edu/
GNU General Public License v3.0
207 stars 89 forks source link

How to debug: Tuning step (step 2 in phrasal.sh) running too long (~10 hours) but empty .binwts file #29

Closed ttpro1995 closed 6 years ago

ttpro1995 commented 6 years ago

Step 2 run too long (for more than 10 hours). I cancel before it finish. It does not generate binwts (empty file). The only binwts file is the file i copy in example folder, which is still empty. I did not see any other .binwts

The process take all CPU and ~ 7GB of RAM (almost all I have) until I cancel. But below is all the logs. No binwts.

CONFIG

.vars

#
# Online parameter tuning with with phrasal-train-tune.sh
#

# General parameters
#
HOST=`hostname -s`
MEM=7g
JAVA_OPTS="-server -ea -Xmx${MEM} -Xms${MEM} -XX:+UseParallelGC -XX:+UseParallelOldGC"
DECODER_OPTS="-Djava.library.path=/home/me/phrasal.ver/src-cc"

# Set if you want to receive an email when a run completes.
# Assumes that the 'mail' unix program is installed and
# configured on your system.
EMAIL_ADDRESS=example@example.com

# Resource locations
#
REFDIR=/data/refdir
CORPUSDIR=/data/corpusdir
CORPUS_SRC=${CORPUSDIR}/train.src.filt.gz
CORPUS_TGT=${CORPUSDIR}/train.dest.filt.gz
CORPUS_EF=${CORPUSDIR}/dest_src.A3.final.merge
CORPUS_FE=${CORPUSDIR}/src_dest.A3.final.merge

# Directory for reporting system.
#REPORTING_DIR=
#RESULTS_FILE=$REPORTING_DIR/results.html

#
# Phrase extraction parameters
#

# Mandatory extraction set format. See Usage of mt.train.PhraseExtract
# for the several different extraction set formats
EXTRACT_SET="-fCorpus $CORPUS_SRC -eCorpus $CORPUS_TGT -feAlign $CORPUS_FE -efAlign $CORPUS_EF -symmetrization grow-diag"
THREADS_EXTRACT=8
MAX_PHRASE_LEN=5
# DEBUG_PROPERTY=true
# DETAILED_DEBUG_PROPERTY=true
OTHER_EXTRACT_OPTS="-phiFilter 1e-4 -maxELen $MAX_PHRASE_LEN"

# Feature extractors
EXTRACTORS=edu.stanford.nlp.mt.train.MosesPharoahFeatureExtractor=phrase-table.gz:edu.stanford.nlp.mt.train.CountFeatureExtractor=phrase-table.gz:edu.stanford.nlp.mt.train.LexicalReorderingFeatureExtractor=lo-hier.msd2-bidirectional-fe.gz
EXTRACTOR_OPTS=""

# Lexicalized re-ordering models
LO_ARGS="-hierarchicalOrientationModel true -orientationModelType msd2-bidirectional-fe"

# Online tuning parameters
TUNE_MODE=online
TUNE_SET_NAME=dev_data
TUNE_SET=$CORPUSDIR/$TUNE_SET_NAME.dest
TUNE_REF=$REFDIR/$TUNE_SET_NAME/ref0
INITIAL_WTS=20171212.binwts
TUNE_NBEST=100

#Options to pass directly to OnlineTuner
METRIC=bleu-smooth
# default
# ONLINE_OPTS="-e 8 -ef 20 -b 20 -uw -m $METRIC -o pro-sgd -of 1,5000,50,0.5,Infinity,0.02,adagradl1f,0.1"
ONLINE_OPTS="-e 1 -ef 10 -b 20 -uw -m $METRIC -o pro-sgd -of 1,5000,50,0.5,Infinity,0.02,adagradl1f,0.1"

# Decoding parameters for dev/test set
DECODE_SET_NAME=test_data
DECODE_SET=$CORPUSDIR/$DECODE_SET_NAME.dest
NBEST=1

.ini

# Example Phrasal ini file
# These options are described by the usage statement
# that is shown on the command line (use the "-help" option).
#
# phrasal.sh will modify this template depending on the steps
# selected to run.
#

# phrasal.sh replaces the token SETID with the
# dev or test set name.
[ttable-file]
SETID.tables/phrase-table.gz

# The 'kenlm:' enables the KenLM loader. Remove the
# prefix for the standard Java ARPA loader.
[lmodel-file]
/data/kenlm.arpa

[ttable-limit]
20

[distortion-limit]
5

# The dense Moses feature set is loaded by default.
# Also load the hierarchical re-ordering model of Galley and Manning (2008)
[reordering-model]
hierarchical
SETID.tables/lo-hier.msd2-bidirectional-fe.gz
msd2-bidirectional-fe
hierarchical
hierarchical
bin

# Number of decoding threads
[threads]
3

LOG

.online.stdout log

Done loading phrase table: /data/dev_data.tables/phrase-table.gz (mem used: 465 MiB time: 0.737 s)
Longest foreign phrase: 5
Loading extended Moses Lexical Reordering Table: dev_data.tables/lo-hier.msd2-bidirectional-fe.gz
Done loading reordering table: dev_data.tables/lo-hier.msd2-bidirectional-fe.gz (mem used: 573 MiB time: 0.716s)
Hierarchical reordering model:
Distinguish between left and right discontinuous: true
Use containment orientation: false
Forward orientation: hierarchical
Backward orientation: hierarchical
Reading 262144 1-grams...
Reading 8388608 2-grams...
Reading 67108864 3-grams...
Reading 134217728 4-grams...
Reading 134217728 5-grams...
Done loading arpa lm: /data/kenlm.arpa (order: 5) (mem used: 4595 MiB time: 172.626 s)

phrasal.log

[INFO ] 2017-12-13 16:37:07.993 [main] OnlineTuner - Phrasal Online Tuner
[INFO ] 2017-12-13 16:37:08.143 [main] OnlineTuner - Options:  /data/dev_data.src /data/refdir/dev_data/ref0 dev_data.20171212baseline.ini 20171212.binwts b 20 e 1 ef 10 m bleu-smooth n dev_data.20171212baseline o pro-sgd of 1,5000,50,0.5,Infinity,0.02,adagradl1f,0.1 uw true
[INFO ] 2017-12-13 16:37:08.174 [main] Phrasal - Number of threads: 8
[INFO ] 2017-12-13 16:37:08.174 [main] Phrasal - Phrase table rule query limit: 20
[INFO ] 2017-12-13 16:37:08.174 [main] Phrasal - Translation model options []
[INFO ] 2017-12-13 16:37:08.949 [main] Phrasal - Translation model mode: static
[INFO ] 2017-12-13 16:37:09.684 [main] Phrasal - Language model: /data/kenlm.arpa
ttpro1995 commented 6 years ago

I found the cause. My Kenlm model is too big (~8GB) which is as big as my RAM.

Training another KenLM model on smaller dataset, yield around 1GB arpa should train faster (in ~ hour for 10 epochs)