POS induction results for the latent crf model

[wammar]

data:

• first 1000 sents in treebank3

features:

• x_i:yi, y{i-1}:y_i (i.e. hmm features)

config:

• 45 latent classes
• L1-regularized CRF objective strength 1.0
• batch lbfgs
• max lbfgs iterations per coordinate descent iteration is 4
• max Evaluate() calls per lbfgs iteration is 3

results:

• variation of information = 3.52891, VMeasure = 0.461545
• many-to-one = 0.51966
• -loglikelihood = 80070.7

runtime:

• runtime = 70 min * 30 cores
• after 10 coordinate descent iterations, relative diff in loglikelihood dropped below 0.01

files:

• example/run1*

[wammar]

features diff:

• add yi:x{i-1}, yi:x{i+1}, y_i:i_small, y_i:prefix(x_i), y_i:suffix(x_i), y_i:hash(x_i), y_i:(|x|-i)_small, y_i:capitalInitial(x_i)And(i>0)

results:

• VI = 3.39924 & VMeasure = 0.477897
• many-to-one = 0.482501
• -loglikelihood = 66521

runtime:

• runtime = 285 min * 30 cores
• after 7 coordinate descent iterations, relative diff in loglikelihood dropped below 0.01

files:

• example/run2*

[wammar]

to contrast: BKP with EM, 45 classes

results:

• VMeasure = 0.574270
• many-to-one = 0.605764

runtime:

• runtime = 17 min * 1 core
• after 1000 EM iterations

[wammar]

feature diff:

• add yi:x{i-2}, yi:x{i+2}, yi:prefix(x{i+1}), yi:suffix(x{i+1}), yi:prefix(x{i-1}), yi:suffix(x{i-1}), y_i:capitalInitial(x_i)

results:

• VI = 3.18732 & VMeasure = 0.483037
• many-to-one = 0.527895
• -loglikelihood = 81622.7

runtime:

• runtime = 11 hours * 28 cores
• after 5 coordinate descent iterations

output files:

• example/run3*

[wammar]

config diff:

• don't regularize

results:

• VI = 3.17293 & VMeasure = 0.50054
• many-to-one = 0.515492
• -loglikelihood = 64363.1

runtime:

• runtime = 10 hours * 30 cores
• after 6 coordinate descent iterations

output files:

• example/run4*

conclusions:

• no surprise. l1 regularization helps.

[wammar]

feature diff:

• remove all prefix features
• fire y_i:suffix(x_i) features for the following suffixes ONLY: -ing, -ogy, -ed, -s, -ly, -ion, -tion, -ity, -ies
• fire capital initial features ONLY when i > 0
• remove the relatively specific hash features
• add coarser hash features
• remove yI:i
• limit yI:(|x|-i) to i = |x| - 1

config diff:

• revert back to using L1 regularizer which turned out to help

results:

• VI = 3.13311 & VMeasure = 0.505113
• many-to-one = 0.526402
• -loglikelihood = 77683.1

runtime:

• runtime = 3 hours * 28 cores
• after 8 coordinate descent iterations

output files:

• run5*

conclusions:

• slightly worse than the best many-to-one result (run3), in about 1/4 the time. it's definitely worth getting rid of some of those unnecessarily specific features.
• the best VI result so far with this model. still far from BKP's results, though.

[wammar]

features diff:

• use only transition features yi:y{i-1}

results:

• VI = 4.40283 & VMeasure = 0.222432
• many-to-one = 0.270715
• -loglikelihood = 67101.7

configuration change:

• gaussian initializer has std deviation of 10 instead of 1

output files:

• example/run6*

[wammar]

feature diff:

• in addition to transition features yi:y{i-1}, also use y_i:x_i

results:

• VI = 4.01088 & VMeasure = 0.404966
• many-to-one = 0.484148
• nloglikelihood=27161.3

output files:

• example/run7*

[wammar]

feature diff:

• insetad of y_i:x_i, use yi:x{i-1} and yi:x{i+1}

results:

• VI = 3.87099 & VMeasure = 0.376391
• many-to-one = 0.45579
• nlikelihood = 53289.9

output files:

• example/run8*

[wammar]

to contrast: PB's character based pyp tagger

results:

• VMeasure = 0.634477
• Many-to-One Accuracy = 0.679002

[wammar]

feature diff:

• add y_i:x_i

results:

• VI = 3.88351 & VMeasure = 0.411599
• many-to-one = 0.49228
• nloglikelihood=50280.1

output files:

• example/run9*

conclusions:

• this feature y_i:x_i helps many-to-one and likelihood but hurts VI a little bit.

[wammar]

feature diff:

• only use yi:x{i-1} and yi:x{i+1} features when those are in the closed vocab of english

results:

• VI = 3.89882 & VMeasure = 0.407534
• many-to-one = 0.471024
• nloglikelihood=37447.9

output files:

• example/run10*

conclusions:

• using yi:x{i-1} and yi:x{i+1} features only when in closed vocab deteriorates both many-to-one (with 2% points) and VI (little bit), and improves liklihood a lot!

[wammar]

features diff:

• add y_i as a feature

results:

• VI = 3.65804 & VMeasure = 0.384969
• many-to-one = 0.418425
• -likelihood = 57437.2

output files:

• example/run11*

conclusion:

• y_i apparently deteriorates many-to-one and improves variation of information.

[wammar]

features diff:

• add capital initial features (where i > 0) and coarse hash features

results:

• VI = 3.30275 & VMeasure = 0.480616
• many-to-one = 0.520226
• -loglikelihood = 34119.9

output files:

• example/run12*

conclusions:

• capital initials (where i>0) and coarse hash features help according to all metrics.

[wammar]

features used:

• yi:x{i-1} and yi:x{i+1} (regardless of being in the closed vocab)
• y_i:x_i
• yi:y{i-1}
• capital initial features (where i > 0)
• coarse hash features

features diff:

• remove y_i features
• uses yi:x{i-1} and yi:x{i+1} regardless of whether they are in closed vocab

results:

• VI = 3.2425 & VMeasure = 0.463855
• many-to-one = 0.520587
• -loglikelihood = 58114.7

output files:

• example/run13*

conclusions:

• this run serves as a "rebaseline" after learning from previous experiments.

[wammar]

config diff:

• in reconstruction model p(z|y), each observation z_i depends on both yi and y{i-1}

results:

• variation of information = 4.78547
• many-to-one = 0.388214
• nloglikelihood=61035.2

output files:

• example/run14*

[wammar]

config diff:

• zI depends on yI only.

features diff:

• add yI-i and yI-(|x|-i) for i, |x|-i < 2

results:

• VI = 3.19381 & VMeasure = 0.48893
• many-to-one = 0.547761
• nloglikelihood = 50712.6

conclusions:

• by contrasting with run13, positional features y_i:i and y_i:(|x|-i) are useful

files: example/run16

[wammar]

to do experiments:

• add yi-prefix(x{i-k}) where k \in {-2, -1, 0, 1, 2}
• add yi-x{i-k} where k \in {-2, 2}
• vary number of classes

[wammar]

hmm baseline:

• 45 classes
• gaussian initialization
• stopping criterion is a relative difference in loglikelihood < 0.001

results:

• VI = 4.931, VMeasure = 0.267
• many-to-one = 0.36001
• nloglikelihood = 102277

files: example/run.hmm.40*

[wammar]

experiment id = 18 job id = 239729.tg-login1.blacklight.psc.teragrid.org

pbs configs: @ run-tb3-pos-induction-blacklight-18.job

PBS -l ncpus=64

PBS -l pmem=50gb

PBS -l walltime=48:00:00

PBS -d .

PBS -j oe

command: mpirun -np 32 ./train-latentCrfModel-18 example/tb3-pos-wsj.eng example/run-tb3-pos-induction-blacklight-18.out example/tb3-pos-wsj.pos &> /usr/users/9/wammar/alignment-with-openfst/example/run-tb3-pos-induction-blacklight-18.log

data:

• all of english treebank3 example/tb3-pos-wsj.{eng|pos}

configs:

• hmm initialization with 100 EM iterations and parallelized random restarts
• 45 latent classes

features:

• yi:y{i-1}
• yi:x{i-1}, y_i:x_i, yi:x{i+1}
• y_i:i, y_i:(|x|-i)
• capital and i != 0
• coarse hash function

[wammar]

experiment id = 19 job id = 239826

pbs configs: @ run-tb3-pos-induction-blacklight-19.job

PBS -l ncpus=64

PBS -l pmem=50gb

PBS -l walltime=48:00:00

PBS -d .

PBS -j oe

command: mpirun -np 32 ./train-latentCrfModel-19 example/tb3-pos-wsj.eng example/run-tb3-pos-induction-blacklight-19.out example/tb3-pos-wsj.pos &> /usr/users/9/wammar/alignment-with-openfst/example/run-tb3-pos-induction-blacklight-19.log

data: top 1000 sents of english treebank3 example/tb3-pos-wsj-1000.{eng|pos}

configs:

• hmm initialization with 100 EM iterations and parallelized random restarts
• 45 latent classes

features:

• yi:y{i-1}
• yi:x{i-1}, y_i:x_i, yi:x{i+1}
• y_i:i, y_i:(|x|-i)
• capital and i != 0
• coarse hash function

results: before hmm init:

• VI = 4.8214 & VMeasure = 0.272558
• likelihood = 101341
• many-to-one = 0.373906

after latent crf training:

• VI = 4.24095 & VMeasure = 0.300471
• likelihood = 56960.8
• many-to-one = 0.373598

[wammar]

experiment id = 20 job id = 239825

pbs config: @ run-medium-pos-induction-blacklight-20.job

data: ~10K kinyarwanda sents

configs:

• hmm initialization with 100 EM iterations and parallelized random restarts
• 45 latent classes

features:

• yi:y{i-1}
• yi:x{i-1}, y_i:x_i, yi:x{i+1}
• y_i:i, y_i:(|x|-i)
• capital and i != 0
• coarse hash function