Step 2 run too long (for more than 10 hours). I cancel before it finish. It does not generate binwts (empty file). The only binwts file is the file i copy in example folder, which is still empty. I did not see any other .binwts
The process take all CPU and ~ 7GB of RAM (almost all I have) until I cancel. But below is all the logs. No binwts.
CONFIG
.vars
#
# Online parameter tuning with with phrasal-train-tune.sh
#
# General parameters
#
HOST=`hostname -s`
MEM=7g
JAVA_OPTS="-server -ea -Xmx${MEM} -Xms${MEM} -XX:+UseParallelGC -XX:+UseParallelOldGC"
DECODER_OPTS="-Djava.library.path=/home/me/phrasal.ver/src-cc"
# Set if you want to receive an email when a run completes.
# Assumes that the 'mail' unix program is installed and
# configured on your system.
EMAIL_ADDRESS=example@example.com
# Resource locations
#
REFDIR=/data/refdir
CORPUSDIR=/data/corpusdir
CORPUS_SRC=${CORPUSDIR}/train.src.filt.gz
CORPUS_TGT=${CORPUSDIR}/train.dest.filt.gz
CORPUS_EF=${CORPUSDIR}/dest_src.A3.final.merge
CORPUS_FE=${CORPUSDIR}/src_dest.A3.final.merge
# Directory for reporting system.
#REPORTING_DIR=
#RESULTS_FILE=$REPORTING_DIR/results.html
#
# Phrase extraction parameters
#
# Mandatory extraction set format. See Usage of mt.train.PhraseExtract
# for the several different extraction set formats
EXTRACT_SET="-fCorpus $CORPUS_SRC -eCorpus $CORPUS_TGT -feAlign $CORPUS_FE -efAlign $CORPUS_EF -symmetrization grow-diag"
THREADS_EXTRACT=8
MAX_PHRASE_LEN=5
# DEBUG_PROPERTY=true
# DETAILED_DEBUG_PROPERTY=true
OTHER_EXTRACT_OPTS="-phiFilter 1e-4 -maxELen $MAX_PHRASE_LEN"
# Feature extractors
EXTRACTORS=edu.stanford.nlp.mt.train.MosesPharoahFeatureExtractor=phrase-table.gz:edu.stanford.nlp.mt.train.CountFeatureExtractor=phrase-table.gz:edu.stanford.nlp.mt.train.LexicalReorderingFeatureExtractor=lo-hier.msd2-bidirectional-fe.gz
EXTRACTOR_OPTS=""
# Lexicalized re-ordering models
LO_ARGS="-hierarchicalOrientationModel true -orientationModelType msd2-bidirectional-fe"
# Online tuning parameters
TUNE_MODE=online
TUNE_SET_NAME=dev_data
TUNE_SET=$CORPUSDIR/$TUNE_SET_NAME.dest
TUNE_REF=$REFDIR/$TUNE_SET_NAME/ref0
INITIAL_WTS=20171212.binwts
TUNE_NBEST=100
#Options to pass directly to OnlineTuner
METRIC=bleu-smooth
# default
# ONLINE_OPTS="-e 8 -ef 20 -b 20 -uw -m $METRIC -o pro-sgd -of 1,5000,50,0.5,Infinity,0.02,adagradl1f,0.1"
ONLINE_OPTS="-e 1 -ef 10 -b 20 -uw -m $METRIC -o pro-sgd -of 1,5000,50,0.5,Infinity,0.02,adagradl1f,0.1"
# Decoding parameters for dev/test set
DECODE_SET_NAME=test_data
DECODE_SET=$CORPUSDIR/$DECODE_SET_NAME.dest
NBEST=1
.ini
# Example Phrasal ini file
# These options are described by the usage statement
# that is shown on the command line (use the "-help" option).
#
# phrasal.sh will modify this template depending on the steps
# selected to run.
#
# phrasal.sh replaces the token SETID with the
# dev or test set name.
[ttable-file]
SETID.tables/phrase-table.gz
# The 'kenlm:' enables the KenLM loader. Remove the
# prefix for the standard Java ARPA loader.
[lmodel-file]
/data/kenlm.arpa
[ttable-limit]
20
[distortion-limit]
5
# The dense Moses feature set is loaded by default.
# Also load the hierarchical re-ordering model of Galley and Manning (2008)
[reordering-model]
hierarchical
SETID.tables/lo-hier.msd2-bidirectional-fe.gz
msd2-bidirectional-fe
hierarchical
hierarchical
bin
# Number of decoding threads
[threads]
3
Step 2 run too long (for more than 10 hours). I cancel before it finish. It does not generate binwts (empty file). The only binwts file is the file i copy in example folder, which is still empty. I did not see any other .binwts
The process take all CPU and ~ 7GB of RAM (almost all I have) until I cancel. But below is all the logs. No binwts.
CONFIG
.vars
.ini
LOG
.online.stdout log
phrasal.log