Closed zhyack closed 5 years ago
Hi, I think the reason is you print the generation before fast adapt. I modified the code that only print the generation after 10 iteration finetune, and you can pull and run main_finetune again. I attached one of the checkpoint. https://drive.google.com/file/d/1QhdWKTByyOW0tUuOBUs1ZKuhHzVjFCFx/view?usp=sharing Let me know if you have any problem
In the main_fine_tune.py code, I evaluate generation with (0, 1, 3, 7, 10) iteration finetune to see the generation change, what you see might be the generation with 0 iteration finetune.
Thank you for your quick reply.
However, there is still something wrong. I want to make sure that the shared checkpoint is exactly for PAML model. I run python main_fine_tune.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weig ht_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/paml/model_1_46.6457_0.0000_0.0000_0.0000_1.1000 --save_path_dataset save/paml/ --test
yesterday and get the final outputs as follows:
['hi , i am , pre law student , only child will finish school in two years .',
'hi i am a high school drop out with three sisters',
'okay . did you and your sister start a tech company or something ?',
'no i am lead singer in a band',
'good for you ! are you copyrighting the music you write ?',
'yes i am are you a parent ?',
'yes ! divorce , male , two children , ex named doug .',
'i home schooled while my dad drove a semi',
'oh okay . are you a country singer ?',
'no i am in a punk grunge band . what do you do ?',
'pre law student . you can use your home schooling in your songs and music .',
'i do . how old are you ?',
'interesting question . do your band have youtube videos posted ?']
Beam: i have have have have have . .
Ref:it does . do you work part time ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
'hi i am a high school drop out with three sisters',
'okay . did you and your sister start a tech company or something ?',
'no i am lead singer in a band',
'good for you ! are you copyrighting the music you write ?',
'yes i am are you a parent ?',
'yes ! divorce , male , two children , ex named doug .',
'i home schooled while my dad drove a semi',
'oh okay . are you a country singer ?',
'no i am in a punk grunge band . what do you do ?',
'pre law student . you can use your home schooling in your songs and music .']
Beam: i have have have have have have have have .
Ref:i do . how old are you ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
'hi i am a high school drop out with three sisters',
'okay . did you and your sister start a tech company or something ?',
'no i am lead singer in a band',
'good for you ! are you copyrighting the music you write ?',
'yes i am are you a parent ?',
'yes ! divorce , male , two children , ex named doug .',
'i home schooled while my dad drove a semi',
'oh okay . are you a country singer ?']
Beam: i have have have have have have have have have have have have
Ref:no i am in a punk grunge band . what do you do ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
'hi i am a high school drop out with three sisters',
'okay . did you and your sister start a tech company or something ?',
'no i am lead singer in a band',
'good for you ! are you copyrighting the music you write ?',
'yes i am are you a parent ?',
'yes ! divorce , male , two children , ex named doug .']
Beam: i have have have have have have have have have have
Ref:i home schooled while my dad drove a semi
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
'hi i am a high school drop out with three sisters',
'okay . did you and your sister start a tech company or something ?',
'no i am lead singer in a band',
'good for you ! are you copyrighting the music you write ?']
Beam: i am am am am am am am am am .
Ref:yes i am are you a parent ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .',
'hi i am a high school drop out with three sisters',
'okay . did you and your sister start a tech company or something ?']
Beam: i have have have have have have have have have .
Ref:no i am lead singer in a band
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi , i am , pre law student , only child will finish school in two years .']
Beam: i am am am am am am am am am .
Ref:hi i am a high school drop out with three sisters
----------------------------------------------------------------------
----------------------------------------------------------------------
EVAL Loss Peplexity Entl_b Bleu_b
test 5.7443 312.4062 -0.29 0.00
100%|████████████████████████████████████████████████████████████████████████████████████████████
████████████████████████████████████████████████████████| 100/100 [2:46:34<00:00, 103.00s/it]----------------------------------------------------------------------
{'Bleu_b': array([0. , 0.00414685, 0.01258532, 0.01582213, 0.01414685,
0.03252327]), 'LOSS': array([8.37976616, 7.20382413, 6.80048908, 6.63588022, 6.53726106,
6.44257096]), 'PPL': array([4985.61464614, 1533.05871467, 1016.61650556, 860.22644238,
780.2358832 , 711.26467966]), 'Entl_b': array([-0.01200325, -0.04439766, -0.09765385, -0.15007588, -0.17548136,
-0.15935802])}
----------------------------------------------------------------------
The beam outputs are not even complete natural language sentences. And the final PPLs are very different from those in training process. According to my experience, it may happens when parameters are wrongly loaded (or part of them are not loaded).
Did you pull the new push before you run?
Yes. I have run it twice, getting similar results.
Hi, I guess it might be the vocabulary mismatch. I test the repo in a new environment, this time should be ok. Please pull the repo and retrain paml and test.
If you want to skip training and test, you can directly check the generation in file paml_generation
I will close this issue for now, if you have other problem, feel free to reopen this issue
Yeah, thank you! It works fine now!
python MAML.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --fix_dialnum_train --pointer_gen --save_path save/paml/
, I trained the model for several hours and it stopped automatically. And I got a good checkpointsave/paml/model_1_50.1856_0.0000_0.0000_0.0000_1.1000
.python main_fine_tune.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/paml/model_1_50.1856_0.0000_0.0000_0.0000_1.1000 --save_path_dataset save/paml/ --test
to have a test. Unexpectedly, I get the following outputs:After several fine-tuning steps (100 iterations?), it stops and it does not seem to get better: