search

tagoyal / sow-reap-paraphrasing

Contains data/code for the paper "Neural Syntactic Preordering for Controlled Paraphrase Generation" (ACL 2020).
77 stars 9 forks source link

how can I make sure the sow model is trained properly? #11

Closed TITC closed 3 years ago

TITC commented 3 years ago

I am following your detailed guidance step by step to training my own sow model, but I not sure what's the criterion of judging the model trained enough?

input: BOS Y was already going on , X . EOS
gt output: BOS X , and Y continued . EOS
BOS Y , X . EOS

input: BOS are notification log entries Y to X EOS
gt output: BOS X is Y to the alert log . EOS
BOS X Y EOS

input: BOS are notification log entries linked Y X EOS
gt output: BOS X is linked Y the alert log . EOS
BOS Y have been been X EOS

dev nll per token: 3.501965
done with batch 6000 / 43264 in epoch 0, loss: 3.734751, time:104
train nll per token : 3.734751 

input: BOS Y group X . EOS
gt output: BOS X , the group had Y region@@ ally limited scope . EOS
BOS X , Y . EOS

input: BOS players of 14 extr@@ ale@@ ague Y X EOS
gt output: BOS X was cooked by players of 14 extra@@ gable Y EOS
BOS X Y EOS

input: BOS Y of 14 extr@@ ale@@ ague teams X EOS
gt output: BOS X was cooked by Y of 14 extra@@ gable teams EOS
BOS X , Y EOS

dev nll per token: 3.332786
done with batch 9000 / 43264 in epoch 0, loss: 3.506840, time:103
train nll per token : 3.506840 

input: BOS Y and X by for possible ca@@ s-@@ ev@@ acs EOS
gt output: BOS Y and X for any ca@@ s-@@ ev@@ ac EOS
BOS Y , and X the european states . EOS

input: BOS take Y and stand by X EOS
gt output: BOS take Y and wait X EOS
BOS tell Y X EOS

input: BOS Y and stand by X EOS
gt output: BOS Y and wait X EOS
BOS but Y , and X EOS

I want to make a tweak version in Chinese, so want to go deeper into your repo. XD

tagoyal commented 3 years ago

For the SOW model, you can look at metrics like perplexity, loss etc. to ensure that the training loss is creasing. But the quality would have to be determined qualitatively by looking at the rearrangements that are outputted by the system and if they make sense.

BTW, since you are working with Chinese, you may have to rework the alignment code, because current code uses some heuristics to align phrases that have been tuned for the English language but may not be ideal for others.

tldr;

  1. check the quality of the SOW training data generated by the create_sow_data.sh
  2. spot check the qual. of the generated outputs from the trained model.