Are these normal results？

[ChorlingLau]

Hi, I got some results by following README and running infill.py with task "control_pos". Each sentence is 64-word-long and doesn't seem to quite match the POS, as shown in the following example.

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
    "House shop found 's House type type usually bad usually French and highly - French family and family upscale as for makes convenient our - cuisine cuisine at at has very views at price very low , fine he non type non he perfect UNK call non call Children family by a Mid family family convenient family family family kids at family at convenient", 
    "called stop family 's many usually usually at high usually diner child highly family French French for family upscale as makes make convenient usually grub at because cuisine non has pub non at one price we at UNK non as they he new perfect UNK many atmosphere House he who UNK Bibimbap who new family convenient UNK family But new at family at convenient", 
    "House . at 's many type type take French usually while and Join friendly serving with and family usually as so tag are serve 's non among fine perfect it UNK we , options he perfect he he who UNK customers who he - - Conveniently new who who most options a - a having quiet view at quiet has at a by by", 
    "House One make 's 's usually usually very family usually French child Is for French and for family tasty highly French family UNK UNK pub cuisine cuisine Cambridge in cuisine has has has a cuisine out is UNK UNK The by A he out by at atmosphere atmosphere UNK who a he UNK most beautiful quiet who family But who kids a kids convenient", 
    "called shop family and many type usually at somewhere when and French of - for French and by upscale low with gave UNK may pub has can at at has has with UNK price good the , we 're restaurant atmosphere who are UNK he UNK UNK he UNK can 're family who a family new family family who a at a by convenient", 
    ...

And I have the same question on other tasks except for "control_length".

Could you please tell me whether these are normal results? If yes, how can I get a decent sentence like you show in paper？ Thanks a lot!

[XiangLi1999]

No... these are not normal results...

Some tips for debugging: when you generate unconditionally, does it look fluent? If it's not fluent, it's probably the Diffusion-LM is not trained well. run the classifier/parser/tagger on some normal sentence, if it seems obviously wrong, then it's the classifier/parser/tagger's problem.

[XiangLi1999]

actually, this reminds me of https://github.com/XiangLi1999/Diffusion-LM/issues/21 -- might be worth checking.

[ChorlingLau]

Thanks for your reply! Perhaps it's because I removed "--learn_emb yes" when training classifier. Looking forward to the latest commit which allows the program to 'learn emb'. (´▽｀)ノ♪

[a-antoniades]

@ChorlingLau were you able to get better results? My unconditional generation with either dataset still seems repetitive and incoherent.

[ChorlingLau]

@ChorlingLau were you able to get better results? My unconditional generation with either dataset still seems repetitive and incoherent.

I got coherent sentences in unconditional generation. And after pulling the newest code I get better results in task 'control_pos' but 'control_tree' still not.

unconditional:

["star value food , Midsummer House . Not far from All Bar One . \n END START near Caf\u00e9 Rouge , is a five star restaurant called The Golden Curry , good for families . \n END START There is a coffee shop close to the river called Fitzbillies . It is a family place . \n END START For cheap Fast food in"]
["You can have UNK a meal , but is not not recommended for kids . \n END START Fitzbillies near the Express by Holiday Inn is cheap and not family friendly with an average rating \n END START The high priced Chinese pub Wildwood holds a high customer rating \n END START The Punter is a family - friendly coffee shop that served Japanese"]
...

control_pos:

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
  'the city centre area near Café Sicilia , The Cambridge customer rating has low customer ratings and is not rated coffee shop . \n END START The Eagle , is a UNK Bar with a UNK UNK UNK UNK you UNK UNK of UNK UNK UNK UNK UNK UNK will stop now of wine UNK such and UNK UNK UNK UNK UNK at a',
  'START Located in riverside near Clare Hall , a cheap coffee shop serving fast food restaurant and is not a coffee shop . Cocum has a customer rating of 5 out of 5 . \n END START The Waterman is a children friendly place in the UNK UNK UNK of over £ 30 , with a UNK 5 out of 5 , UNK a',
  'START A coffee shop near Café Sicilia , The Dumpling Tree shop serves moderate price range and is not the city centre . It is not kid friendly . \n END START The Waterman is a restaurant on are UNK UNK UNK , UNK of UNK fresh UNK UNK , while have UNK UNK UNK . They do UNK you UNK UNK UNK UNK',
  'START Moderately priced highly near Café Sicilia , The Punter coffee shop has low customer ratings and is not family - shop . \n END START The Golden Curry is a just UNK the UNK of Cambridge UNK will UNK with the UNK . your children will UNK UNK . UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK',
...

[a-antoniades]

@ChorlingLau hmm, to me that seems like "barely coherent" - but admittedly better than what I'm getting. This is using the commands in the readme for training and decoding right?

@XiangLi1999 any thoughts on the quality of the outputs we've been getting? Seems like it's way off from your results, what could we be doing wrong? Have you used this actual codebase/data to train a model yourself? If so, what are your results/hyperparams? Thanks a lot.

[a-antoniades]

update: i was able to train a much better model that generates nice text, using much larger batch sizes (640).

["area . It is friendly for families . \n END START The Rice Boat serves English food near Express by Holiday Inn in city centre it is family - friendly with a high rating . \n END START There is a restaurant called Aromi that serves Chinese food and is located in the city centre . \n END START For a kid friendly Chinese"]
["Waterman located near Crowne Plaza Hotel . It is not a family friendly restaurant . \n END START Fitzbillies is near Express by Holiday Inn . It is not family - friendly and has a more than \u00a3 30 price range and a 5 out of 5 customer rating \n END START The Punter is a family - friendly coffee shop offering Italian food"]
["riverside . \n END START Fitzbillies is a best cheap Indian restaurant near The Six Bells . \n END START The Waterman is a non family - friendly fast food restaurant with a low customer rating and price range less than 20 euros and is in the city centre . \n END START There is a coffee shop called Fitzbillies located in the city"]

[lcy5058]

感谢您的回复！ 也许是因为我在训练分类器时删除了“--learn_emb yes”。 期待允许程序“学习 emb”的最新提交。(´▽｀)ノ♪

@ChorlingLau你能得到更好的结果吗？我使用任一数据集的无条件生成似乎仍然重复且不连贯。

我在无条件生成中得到了连贯的句子。在提取最新代码后，我在任务“control_pos”中得到了更好的结果，但“control_tree”仍然没有。

无条件：

["star value food , Midsummer House . Not far from All Bar One . \n END START near Caf\u00e9 Rouge , is a five star restaurant called The Golden Curry , good for families . \n END START There is a coffee shop close to the river called Fitzbillies . It is a family place . \n END START For cheap Fast food in"]
["You can have UNK a meal , but is not not recommended for kids . \n END START Fitzbillies near the Express by Holiday Inn is cheap and not family friendly with an average rating \n END START The high priced Chinese pub Wildwood holds a high customer rating \n END START The Punter is a family - friendly coffee shop that served Japanese"]
...

控制位置：

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
  'the city centre area near Café Sicilia , The Cambridge customer rating has low customer ratings and is not rated coffee shop . \n END START The Eagle , is a UNK Bar with a UNK UNK UNK UNK you UNK UNK of UNK UNK UNK UNK UNK UNK will stop now of wine UNK such and UNK UNK UNK UNK UNK at a',
  'START Located in riverside near Clare Hall , a cheap coffee shop serving fast food restaurant and is not a coffee shop . Cocum has a customer rating of 5 out of 5 . \n END START The Waterman is a children friendly place in the UNK UNK UNK of over £ 30 , with a UNK 5 out of 5 , UNK a',
  'START A coffee shop near Café Sicilia , The Dumpling Tree shop serves moderate price range and is not the city centre . It is not kid friendly . \n END START The Waterman is a restaurant on are UNK UNK UNK , UNK of UNK fresh UNK UNK , while have UNK UNK UNK . They do UNK you UNK UNK UNK UNK',
  'START Moderately priced highly near Café Sicilia , The Punter coffee shop has low customer ratings and is not family - shop . \n END START The Golden Curry is a just UNK the UNK of Cambridge UNK will UNK with the UNK . your children will UNK UNK . UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK',
...

I want to know your detailed commands in the last step of running scripts/infill.py and how to generate unconditionally. Thank you very much

[XiangLi1999]

I think one important detail is to train a PAD based model via "--padding_mode pad ". Essentially, this makes a sentence start from [START] [sentence content] [END] [PAD*n]. Training by block looks fine for unconditional generation, but for controls, I think there are too much noise when learning the classifier.

[ChorlingLau]

感谢您的回复！ 也许是因为我在训练分类器时删除了“--learn_emb yes”。 期待允许程序“学习 emb”的最新提交。(´▽｀)ノ♪

@ChorlingLau你能得到更好的结果吗？我使用任一数据集的无条件生成似乎仍然重复且不连贯。

我在无条件生成中得到了连贯的句子。在提取最新代码后，我在任务“control_pos”中得到了更好的结果，但“control_tree”仍然没有。 无条件：

["star value food , Midsummer House . Not far from All Bar One . \n END START near Caf\u00e9 Rouge , is a five star restaurant called The Golden Curry , good for families . \n END START There is a coffee shop close to the river called Fitzbillies . It is a family place . \n END START For cheap Fast food in"]
["You can have UNK a meal , but is not not recommended for kids . \n END START Fitzbillies near the Express by Holiday Inn is cheap and not family friendly with an average rating \n END START The high priced Chinese pub Wildwood holds a high customer rating \n END START The Punter is a family - friendly coffee shop that served Japanese"]
...

控制位置：

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
  'the city centre area near Café Sicilia , The Cambridge customer rating has low customer ratings and is not rated coffee shop . \n END START The Eagle , is a UNK Bar with a UNK UNK UNK UNK you UNK UNK of UNK UNK UNK UNK UNK UNK will stop now of wine UNK such and UNK UNK UNK UNK UNK at a',
  'START Located in riverside near Clare Hall , a cheap coffee shop serving fast food restaurant and is not a coffee shop . Cocum has a customer rating of 5 out of 5 . \n END START The Waterman is a children friendly place in the UNK UNK UNK of over £ 30 , with a UNK 5 out of 5 , UNK a',
  'START A coffee shop near Café Sicilia , The Dumpling Tree shop serves moderate price range and is not the city centre . It is not kid friendly . \n END START The Waterman is a restaurant on are UNK UNK UNK , UNK of UNK fresh UNK UNK , while have UNK UNK UNK . They do UNK you UNK UNK UNK UNK',
  'START Moderately priced highly near Café Sicilia , The Punter coffee shop has low customer ratings and is not family - shop . \n END START The Golden Curry is a just UNK the UNK of Cambridge UNK will UNK with the UNK . your children will UNK UNK . UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK',
...

I want to know your detailed commands in the last step of running scripts/infill.py and how to generate unconditionally. Thank you very much

# unconditional
scripts/batch_decode.py [diffusion model path] -1.0 ema

# controlled
scripts/infill.py --model_path [path]/model200000.pt --eval_task_ 'control_pos' --use_ddim True --notes "pos_adagrad" --eta 1. --verbose pipe

[lcy5058]

感谢您的回复！ 也许是因为我在训练分类器时删除了“--learn_emb yes”。 期待允许程序“学习 emb”的最新提交。(´▽｀)ノ♪

@ChorlingLau你能得到更好的结果吗？我使用任一数据集的无条件生成似乎仍然重复且不连贯。

我在无条件生成中得到了连贯的句子。在提取最新代码后，我在任务“control_pos”中得到了更好的结果，但“control_tree”仍然没有。 无条件：

["star value food , Midsummer House . Not far from All Bar One . \n END START near Caf\u00e9 Rouge , is a five star restaurant called The Golden Curry , good for families . \n END START There is a coffee shop close to the river called Fitzbillies . It is a family place . \n END START For cheap Fast food in"]
["You can have UNK a meal , but is not not recommended for kids . \n END START Fitzbillies near the Express by Holiday Inn is cheap and not family friendly with an average rating \n END START The high priced Chinese pub Wildwood holds a high customer rating \n END START The Punter is a family - friendly coffee shop that served Japanese"]
...

控制位置：

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
  'the city centre area near Café Sicilia , The Cambridge customer rating has low customer ratings and is not rated coffee shop . \n END START The Eagle , is a UNK Bar with a UNK UNK UNK UNK you UNK UNK of UNK UNK UNK UNK UNK UNK will stop now of wine UNK such and UNK UNK UNK UNK UNK at a',
  'START Located in riverside near Clare Hall , a cheap coffee shop serving fast food restaurant and is not a coffee shop . Cocum has a customer rating of 5 out of 5 . \n END START The Waterman is a children friendly place in the UNK UNK UNK of over £ 30 , with a UNK 5 out of 5 , UNK a',
  'START A coffee shop near Café Sicilia , The Dumpling Tree shop serves moderate price range and is not the city centre . It is not kid friendly . \n END START The Waterman is a restaurant on are UNK UNK UNK , UNK of UNK fresh UNK UNK , while have UNK UNK UNK . They do UNK you UNK UNK UNK UNK',
  'START Moderately priced highly near Café Sicilia , The Punter coffee shop has low customer ratings and is not family - shop . \n END START The Golden Curry is a just UNK the UNK of Cambridge UNK will UNK with the UNK . your children will UNK UNK . UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK',
...

I want to know your detailed commands in the last step of running scripts/infill.py and how to generate unconditionally. Thank you very much

# unconditional
scripts/batch_decode.py [diffusion model path] -1.0 ema

# controlled
scripts/infill.py --model_path [path]/model200000.pt --eval_task_ 'control_pos' --use_ddim True --notes "pos_adagrad" --eta 1. --verbose pipe

Thanks,When finally generating controllable text, python scripts/infill.py --model_path {path-to-diffusion-lm} command, which path does "path-to-diffusion-lm" refer to, the first step model path, or the last but one model generated from the classifier? "improved diffusion/diffusion_models/diff_e2e-tgt_block_rand16_transformer_lr0.0001_0.0_2000_sqrt_Lsimple_h128_s2_d0.1_sd102_xstart_e2e" or "Classifier_models/e2e-tgt-tree_e=6_b=10_m=bert base uncased_wikitext-103-raw-v1_101_wp_None"

[ChorlingLau]

@Licy1999 It's 1st step model. The classifier path should be modified in code file infill.py

[lcy5058]

@Licy1999 It's 1st step model. The classifier path should be modified in code file infill.py

hello,I am filling as you said infill.py changes the training path of the classifier model, code:model_control = Classifier_POS.from_pretrained('../classifier_models/e2e-tgt-tree_e=6_b=10_m=bert-base-uncased_wikitext-103-raw-v1_101_wp_None').cuda(),and I confirm the file of oil training classifier in this folder.but it seems that the loading fails. An error like this is reported. Can you tell me how to handle it. load the partial sequences None Traceback (most recent call last): File "scripts/infill.py", line 764, in args = main() File "scripts/infill.py", line 241, in main model_control = Classifier_POS.from_pretrained('../classifier_models/e2e-tgt-tree_e=6_b=10_m=bert-base-uncased_wikitext-103-raw-v1_101_wp_None').cuda() File "/tmp/pycharm_project_305/transformers/src/transformers/modeling_utils.py", line 1496, in from_pretrained model = cls(config, *model_args, **model_kwargs) File "/tmp/pycharm_project_25/improved-diffusion/../transformers/examples/pytorch/language-modeling/custom_trainer.py", line 1444, in init self.lm_head2 = nn.Linear(config.hidden_size, config.pos_vocab_size, bias=False) File "/tmp/pycharm_project_305/transformers/src/transformers/configuration_utils.py", line 252, in getattribute return super().getattribute(key) AttributeError: 'BertConfig' object has no attribute 'pos_vocab_size'