Need help on reproducing the results.

[zhyack]

• I followed the instructions in README to build up the environments and successfully got the codes run.
• Using the script python MAML.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --fix_dialnum_train --pointer_gen --save_path save/paml/, I trained the model for several hours and it stopped automatically. And I got a good checkpoint save/paml/model_1_50.1856_0.0000_0.0000_0.0000_1.1000.
• And I run python main_fine_tune.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/paml/model_1_50.1856_0.0000_0.0000_0.0000_1.1000 --save_path_dataset save/paml/ --test to have a test. Unexpectedly, I get the following outputs:

  ['hi there how are you doing ?',
  'i am doing good . just enjoying eating my muffin , you ?',
  'i am well thank you just waiting for my kiddos to get home from school . '
  'than dinner',
  'i wish i had kids . just me and my dog , pedro , here',
  'give it time dear . they are a lot of work and well worth the wait',
  'i work with a lot of kids in the healthcare industry',
  'that is a noble profession how did you get into that',
  'i originally wanted to have a career in film as i love it',
  'what exactly do you want to do in that field ?',
  'produce . i decided to go down the healthcare route because its hard to get '
  'into',
  'i can see that do you have a starting place ?',
  'i am not sure yet . so , what do you like ?',
  'reading is a passion , as well as cooking for my family . pretty simple here',
  'sounds like a nice life you got',
  'i think it is . everyday life with little drama from outsiders just how i '
  'like it']
  Beam: i am a i am i am as as as as as as as ?
  Ref:well , i am happy for you !
  ----------------------------------------------------------------------
  ----------------------------------------------------------------------
  ['hi there how are you doing ?',
  'i am doing good . just enjoying eating my muffin , you ?',
  'i am well thank you just waiting for my kiddos to get home from school . '
  'than dinner',
  'i wish i had kids . just me and my dog , pedro , here',
  'give it time dear . they are a lot of work and well worth the wait',
  'i work with a lot of kids in the healthcare industry',
  'that is a noble profession how did you get into that',
  'i originally wanted to have a career in film as i love it',
  'what exactly do you want to do in that field ?',
  'produce . i decided to go down the healthcare route because its hard to get '
  'into',
  'i can see that do you have a starting place ?',
  'i am not sure yet . so , what do you like ?',
  'reading is a passion , as well as cooking for my family . pretty simple here']
  Beam: i am as as as as as as as as as as as as as as as
  Ref:sounds like a nice life you got
  ----------------------------------------------------------------------
  ----------------------------------------------------------------------
  ['hi there how are you doing ?',
  'i am doing good . just enjoying eating my muffin , you ?',
  'i am well thank you just waiting for my kiddos to get home from school . '
  'than dinner',
  'i wish i had kids . just me and my dog , pedro , here',
  'give it time dear . they are a lot of work and well worth the wait',
  'i work with a lot of kids in the healthcare industry',
  'that is a noble profession how did you get into that',
  'i originally wanted to have a career in film as i love it',
  'what exactly do you want to do in that field ?',
  'produce . i decided to go down the healthcare route because its hard to get '
  'into',
  'i can see that do you have a starting place ?']
  Beam: i am ? i am i am i am i am as as as as
  Ref:i am not sure yet . so , what do you like ?
  ----------------------------------------------------------------------
  ----------------------------------------------------------------------
  ['hi there how are you doing ?',
  'i am doing good . just enjoying eating my muffin , you ?',
  'i am well thank you just waiting for my kiddos to get home from school . '
  'than dinner',
  'i wish i had kids . just me and my dog , pedro , here',
  'give it time dear . they are a lot of work and well worth the wait',
  'i work with a lot of kids in the healthcare industry',
  'that is a noble profession how did you get into that',
  'i originally wanted to have a career in film as i love it',
  'what exactly do you want to do in that field ?']
  Beam: i am i am as as as as as as as as as as as as as
  Ref:produce . i decided to go down the healthcare route because its hard to get into
  ----------------------------------------------------------------------
  ----------------------------------------------------------------------
  ['hi there how are you doing ?',
  'i am doing good . just enjoying eating my muffin , you ?',
  'i am well thank you just waiting for my kiddos to get home from school . '
  'than dinner',
  'i wish i had kids . just me and my dog , pedro , here',
  'give it time dear . they are a lot of work and well worth the wait',
  'i work with a lot of kids in the healthcare industry',
  'that is a noble profession how did you get into that']
  Beam: i am ? i am i am i am i am i am i am series
  Ref:i originally wanted to have a career in film as i love it
  ----------------------------------------------------------------------
  ----------------------------------------------------------------------
  ['hi there how are you doing ?',
  'i am doing good . just enjoying eating my muffin , you ?',
  'i am well thank you just waiting for my kiddos to get home from school . '
  'than dinner',
  'i wish i had kids . just me and my dog , pedro , here',
  'give it time dear . they are a lot of work and well worth the wait']
  Beam: i am ? i am a i am i am i am i am i am acting
  Ref:i work with a lot of kids in the healthcare industry
  ----------------------------------------------------------------------
  ----------------------------------------------------------------------
  ['hi there how are you doing ?',
  'i am doing good . just enjoying eating my muffin , you ?',
  'i am well thank you just waiting for my kiddos to get home from school . '
  'than dinner']
  Beam: i am ? i am i am i am i am as as as as
  Ref:i wish i had kids . just me and my dog , pedro , here
  ----------------------------------------------------------------------
  ----------------------------------------------------------------------
  ['hi there how are you doing ?']
  Beam: i am pass especially a i am especially a i am do half the
  Ref:i am doing good . just enjoying eating my muffin , you ?
  ----------------------------------------------------------------------
  ----------------------------------------------------------------------
  EVAL    Loss    Peplexity   Entl_b  Bleu_b
  test    6.6667  785.7816    0.00    0.00

After several fine-tuning steps (100 iterations?), it stops and it does not seem to get better:

----------------------------------------------------------------------
{'Bleu_b': array([0.        , 0.00462254, 0.        , 0.00353671, 0.0037849 ,
       0.00433299]), 'Entl_b': array([-0.10235387, -0.02899247, -0.00670951,  0.00621429,  0.01460281,
        0.01853968]), 'PPL': array([6289.55504171, 1681.21756928, 1077.9117038 ,  916.79228123,
        836.58960702,  767.79717897]), 'LOSS': array([8.62963292, 7.31024756, 6.876336  , 6.71580096, 6.62371861,
       6.53619826])}
----------------------------------------------------------------------

• I haven't gone deeper for the codes. But I'd like to build up your dialog system quickly and do some brainstorms after observation. Could you please help me find out the problems in my reproduction?
• And will you share your checkpoints to reproduce the results. It will be great!

[zlinao]

Hi, I think the reason is you print the generation before fast adapt. I modified the code that only print the generation after 10 iteration finetune, and you can pull and run main_finetune again. I attached one of the checkpoint. https://drive.google.com/file/d/1QhdWKTByyOW0tUuOBUs1ZKuhHzVjFCFx/view?usp=sharing Let me know if you have any problem

[zlinao]

In the main_fine_tune.py code, I evaluate generation with (0, 1, 3, 7, 10) iteration finetune to see the generation change, what you see might be the generation with 0 iteration finetune.

[zhyack]

Thank you for your quick reply. However, there is still something wrong. I want to make sure that the shared checkpoint is exactly for PAML model. I run python main_fine_tune.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weig ht_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/paml/model_1_46.6457_0.0000_0.0000_0.0000_1.1000 --save_path_dataset save/paml/ --test yesterday and get the final outputs as follows:

['hi , i am , pre law student , only child will finish school in two years .',
 'hi i am a high school drop out with three sisters',
 'okay . did you and your sister start a tech company or something ?',
 'no i am lead singer in a band',
 'good for you ! are you copyrighting the music you write ?',
 'yes i am are you a parent ?',
 'yes ! divorce , male , two children , ex named doug .',
 'i home schooled while my dad drove a semi',
 'oh okay . are you a country singer ?',
 'no i am in a punk grunge band . what do you do ?',
 'pre law student . you can use your home schooling in your songs and music .',
 'i do . how old are you ?',
 'interesting question . do your band have youtube videos posted ?']
Beam: i have have have have have . .
Ref:it does . do you work part time ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
 'hi i am a high school drop out with three sisters',
 'okay . did you and your sister start a tech company or something ?',
 'no i am lead singer in a band',
 'good for you ! are you copyrighting the music you write ?',
 'yes i am are you a parent ?',
 'yes ! divorce , male , two children , ex named doug .',
 'i home schooled while my dad drove a semi',
 'oh okay . are you a country singer ?',
 'no i am in a punk grunge band . what do you do ?',
 'pre law student . you can use your home schooling in your songs and music .']
Beam: i have have have have have have have have .
Ref:i do . how old are you ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
 'hi i am a high school drop out with three sisters',
 'okay . did you and your sister start a tech company or something ?',
 'no i am lead singer in a band',
 'good for you ! are you copyrighting the music you write ?',
 'yes i am are you a parent ?',
 'yes ! divorce , male , two children , ex named doug .',
 'i home schooled while my dad drove a semi',
 'oh okay . are you a country singer ?']
Beam: i have have have have have have have have have have have have
Ref:no i am in a punk grunge band . what do you do ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
 'hi i am a high school drop out with three sisters',
 'okay . did you and your sister start a tech company or something ?',
 'no i am lead singer in a band',
 'good for you ! are you copyrighting the music you write ?',
 'yes i am are you a parent ?',
 'yes ! divorce , male , two children , ex named doug .']
Beam: i have have have have have have have have have have
Ref:i home schooled while my dad drove a semi
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
 'hi i am a high school drop out with three sisters',
 'okay . did you and your sister start a tech company or something ?',
 'no i am lead singer in a band',
 'good for you ! are you copyrighting the music you write ?']
Beam: i am am am am am am am am am .
Ref:yes i am are you a parent ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
 'hi i am a high school drop out with three sisters',
 'okay . did you and your sister start a tech company or something ?']
Beam: i have have have have have have have have have .
Ref:no i am lead singer in a band
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .']
Beam: i am am am am am am am am am .
Ref:hi i am a high school drop out with three sisters
----------------------------------------------------------------------
----------------------------------------------------------------------
EVAL    Loss    Peplexity   Entl_b  Bleu_b
test    5.7443  312.4062    -0.29   0.00
100%|████████████████████████████████████████████████████████████████████████████████████████████
████████████████████████████████████████████████████████| 100/100 [2:46:34<00:00, 103.00s/it]----------------------------------------------------------------------
{'Bleu_b': array([0.        , 0.00414685, 0.01258532, 0.01582213, 0.01414685,
       0.03252327]), 'LOSS': array([8.37976616, 7.20382413, 6.80048908, 6.63588022, 6.53726106,
       6.44257096]), 'PPL': array([4985.61464614, 1533.05871467, 1016.61650556,  860.22644238,
        780.2358832 ,  711.26467966]), 'Entl_b': array([-0.01200325, -0.04439766, -0.09765385, -0.15007588, -0.17548136,
       -0.15935802])}
----------------------------------------------------------------------

The beam outputs are not even complete natural language sentences. And the final PPLs are very different from those in training process. According to my experience, it may happens when parameters are wrongly loaded (or part of them are not loaded).

[zlinao]

Did you pull the new push before you run?

[zhyack]

Yes. I have run it twice, getting similar results.

[zlinao]

Hi, I guess it might be the vocabulary mismatch. I test the repo in a new environment, this time should be ok. Please pull the repo and retrain paml and test.

If you want to skip training and test, you can directly check the generation in file paml_generation

[zlinao]

I will close this issue for now, if you have other problem, feel free to reopen this issue

[zhyack]

Yeah, thank you! It works fine now!