About the result of the eval.py HS, and the best results

Hi, Lyu, I read your paper Embedding API dependency graph for neural code generation and I learned a lot from it. I downloaded the full dataset and ran the code in your GitHub repository as you suggested. I use Google Colab to run this project. I first ran python3 training.py [HS/MTG] and got the model. Then I ran python3 eval.py [HS/MTG] to evaluate the model results. However, I encountered two problems. 1. The output of running the eval.py HS according to the source code seems to have a little problem, the output of each one seems to be the same and the blue score is very low. Take the HS dataset, for example, the outputs of the HS dataset are as follows.

graph establishing... 100% 200/200 [00:00<00:00, 21650.81it/s] evaluation begin... 1 1 3 5 5 6 7 /content/drive/MyDrive/ADG/ADG-Seq2Seq-master/model_tools.py:262: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requiresgrad(True), rather than torch.tensor(sourceTensor). decoder_input = torch.tensor(node_onehot_t[K][token], dtype=torch.float32).to(device) 9 7 9 9 7 8 9 7 9 9 7 9 7 7 9 /usr/local/lib/python3.7/dist-packages/nltk/translate/bleu_score.py:490: UserWarning: Corpus/Sentence contains 0 counts of 3-gram overlaps. BLEU scores might be undesirable; use SmoothingFunction(). warnings.warn(_msg) input: _GO Enhance o Mechano NAME_END 3 ATK_END 2 DEF_END 4 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END Mech RACE_END Epic RARITY_END b Battlecry b Give your other minions b Windfury b b Taunt b or b Divine Shield b NL i at random i _EOS output: _GO _GO class MinionCard MinionCard def super super super ground truth: _GO class EnhanceoMechano MinionCard def init self super init Enhance o Mechano 4 CHARACTER_CLASS ALL CARD_RARITY EPIC minion_type MINION_TYPE MECH battlecry Battlecry Give Buff Windfury Buff Taunt Buff DivineShield RandomPicker MinionSelector def create_minion self player return Minion 3 2 _EOS BLEU: 0.01638725

1 1 3 5 5 6 7 9 7 9 9 7 8 9 9 9 7 7 7 9 6 7 /usr/local/lib/python3.7/dist-packages/nltk/translate/bleu_score.py:490: UserWarning: Corpus/Sentence contains 0 counts of 4-gram overlaps. BLEU scores might be undesirable; use SmoothingFunction(). warnings.warn(_msg) input: _GO Explosive Sheep NAME_END 1 ATK_END 1 DEF_END 2 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END Mech RACE_END Common RARITY_END b Deathrattle b Deal 2 damage to all minions _EOS output: _GO _GO class MinionCard def init super ground truth: _GO class ExplosiveSheep MinionCard def init self super init Explosive Sheep 2 CHARACTER_CLASS ALL CARD_RARITY COMMON minion_type MINION_TYPE MECH def create_minion self player return Minion 1 1 deathrattle Deathrattle Damage 2 CharacterSelector players BothPlayer taunt True _EOS BLEU: 0.00642959

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 7 9 7 9 input: _GO Feign Death NAME_END 1 ATK_END 1 DEF_END 2 COST_END 1 DUR_END Spell TYPE_END Hunter PLAYER_CLS_END NIL RACE_END Epic RARITY_END Trigger all b Deathrattles b on your minions _EOS output: _GO _GO class MinionCard MinionCard def init super super super ground truth: _GO class FeignDeath SpellCard def init self super init Feign Death 2 CHARACTER_CLASS HUNTER CARD_RARITY EPIC def use self player game super use player game for minion in sorted player minions key lambda m m born for deathrattle in minion deathrattle deathrattle do minion if player double_deathrattle for deathrattle in minion deathrattle deathrattle do minion _EOS BLEU: 0.00510767

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 7 9 7 9 input: _GO Fel Cannon NAME_END 3 ATK_END 5 DEF_END 4 COST_END 1 DUR_END Minion TYPE_END Warlock PLAYER_CLS_END Mech RACE_END Rare RARITY_END At the end of your turn deal 2 damage to a non Mech minion _EOS output: _GO _GO class MinionCard MinionCard def init super super super ground truth: _GO class FelCannon MinionCard def init self super init Fel Cannon 4 CHARACTER_CLASS WARLOCK CARD_RARITY RARE minion_type MINION_TYPE MECH def create_minion self player return Minion 3 5 effects Effect TurnEnded ActionTag Damage 2 MinionSelector Not IsType MINION_TYPE MECH True BothPlayer RandomPicker _EOS BLEU: 0.01464603

1 1 3 5 5 6 7 9 7 9 9 7 8 9 9 9 7 7 7 9 9 7 input: _GO Fel Reaver NAME_END 8 ATK_END 8 DEF_END 5 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END Mech RACE_END Epic RARITY_END Whenever your opponent plays a card remove the top 3 cards of your deck _EOS output: _GO _GO class MinionCard MinionCard def init super super super ground truth: _GO class FelReaver MinionCard def init self super init Fel Reaver 5 CHARACTER_CLASS ALL CARD_RARITY EPIC minion_type MINION_TYPE MECH def create_minion self player return Minion 8 8 effects Effect CardPlayed player EnemyPlayer ActionTag Discard 3 query CardQuery source CARD_SOURCE MY_DECK PlayerSelector _EOS BLEU: 0.01464603

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 9 7 7 9 /usr/local/lib/python3.7/dist-packages/nltk/translate/bleu_score.py:490: UserWarning: Corpus/Sentence contains 0 counts of 2-gram overlaps. BLEU scores might be undesirable; use SmoothingFunction(). warnings.warn(_msg) input: _GO Finicky Cloakfield NAME_END 1 ATK_END 1 DEF_END 1 COST_END 1 DUR_END Spell TYPE_END Neutral PLAYER_CLS_END NIL RACE_END NIL RARITY_END Give a friendly minion b Stealth b until your next turn _EOS output: _GO _GO class MinionCard MinionCard def super super super ground truth: _GO class FinickyCloakfield SpellCard def init self super init Finicky Cloakfield 1 CHARACTER_CLASS ALL CARD_RARITY COMMON False target_func hearthbreaker targeting find_friendly_minion_spell_target def use self player game super use player game self target add_buff BuffUntil Stealth TurnStarted _EOS BLEU: 0.03846303

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 6 7 9 7 input: _GO Flame Leviathan NAME_END 7 ATK_END 7 DEF_END 7 COST_END 1 DUR_END Minion TYPE_END Mage PLAYER_CLS_END Mech RACE_END Legendary RARITY_END When you draw this deal 2 damage to all characters _EOS output: _GO _GO class MinionCard MinionCard def init self super super ground truth: _GO class FlameLeviathan MinionCard def init self super init Flame Leviathan 7 CHARACTER_CLASS MAGE CARD_RARITY LEGENDARY minion_type MINION_TYPE MECH effects Effect Drawn ActionTag Damage 2 CharacterSelector None BothPlayer def create_minion self player return Minion 7 7 _EOS BLEU: 0.02871604

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 9 7 7 9 input: _GO Flamecannon NAME_END 1 ATK_END 1 DEF_END 2 COST_END 1 DUR_END Spell TYPE_END Mage PLAYER_CLS_END NIL RACE_END Common RARITY_END Deal 4 damage to a random enemy minion _EOS output: _GO _GO class MinionCard MinionCard def init super super super ground truth: _GO class Flamecannon SpellCard def init self super init Flamecannon 2 CHARACTER_CLASS MAGE CARD_RARITY COMMON def use self player game super use player game targets hearthbreaker targeting find_enemy_minion_battlecry_target player game lambda x True target game random_choice targets target damage player effective_spell_damage 4 self def can_use self player game return super can_use player game and len game other_player minions 1 _EOS BLEU: 0.00342377

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 7 9 9 9 7 7 9 input: _GO Foe Reaper 4000 NAME_END 6 ATK_END 9 DEF_END 8 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END Mech RACE_END Legendary RARITY_END Also damages the minions next to whomever he attacks _EOS output: _GO _GO class MinionCard MinionCard def super super super ground truth: _GO class FoeReaper4000 MinionCard def init self super init Foe Reaper 4000 8 CHARACTER_CLASS ALL CARD_RARITY LEGENDARY minion_type MINION_TYPE MECH def create_minion self player return Minion 6 9 effects Effect Attack IsMinion ActionTag Damage Attribute attack SelfSelector MinionSelector TargetAdjacent EnemyPlayer _EOS BLEU: 0.01466396

1 1 3 5 5 6 7 9 7 9 9 7 8 9 9 9 7 6 7 9 7 7 input: _GO Force Tank MAX NAME_END 7 ATK_END 7 DEF_END 8 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END Mech RACE_END Common RARITY_END b Divine Shield b _EOS output: _GO _GO class MinionCard MinionCard def init self super super ground truth: _GO class ForceTankMAX MinionCard def init self super init Force Tank MAX 8 CHARACTER_CLASS ALL CARD_RARITY COMMON minion_type MINION_TYPE MECH def create_minion self player return Minion 7 7 divine_shield True _EOS BLEU: 0.05232404

1 1 3 5 5 6 7 9 7 9 9 7 8 9 9 9 7 7 9 7 7 9 input: _GO Gahz rilla NAME_END 6 ATK_END 9 DEF_END 7 COST_END 1 DUR_END Minion TYPE_END Hunter PLAYER_CLS_END Beast RACE_END Legendary RARITY_END Whenever this minion takes damage double its Attack _EOS output: _GO _GO class MinionCard MinionCard def init super super super ground truth: _GO class Gahzrilla MinionCard def init self super init Gahz rilla 7 CHARACTER_CLASS HUNTER CARD_RARITY LEGENDARY minion_type MINION_TYPE BEAST def create_minion self player return Minion 6 9 effects Effect Damaged ActionTag Give Buff DoubleAttack SelfSelector _EOS BLEU: 0.02668680

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 6 7 9 7 input: _GO Gallywix s Coin NAME_END 1 ATK_END 1 DEF_END 0 COST_END 1 DUR_END Spell TYPE_END Neutral PLAYER_CLS_END NIL RACE_END NIL RARITY_END Gain 1 Mana Crystal this turn only NL i Won t trigger Gallywix i _EOS output: _GO _GO class MinionCard MinionCard def super init ground truth: _GO class GallywixsCoin SpellCard def init self super init Gallywix s Coin 0 CHARACTER_CLASS ALL CARD_RARITY COMMON False def use self player game super use player game if player mana 10 player mana 1 _EOS BLEU: 0.02877389

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 9 7 7 9 input: _GO Gazlowe NAME_END 3 ATK_END 6 DEF_END 6 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END NIL RACE_END Legendary RARITY_END Whenever you cast a 1 mana spell add a random Mech to your hand _EOS output: _GO _GO class MinionCard MinionCard def super super super ground truth: _GO class Gazlowe MinionCard def init self super init Gazlowe 6 CHARACTER_CLASS ALL CARD_RARITY LEGENDARY def create_minion self player return Minion 3 6 effects Effect SpellCast ManaCost 1 ActionTag AddCard CardQuery conditions IsType MINION_TYPE MECH PlayerSelector _EOS BLEU: 0.02287025

1 1 3 5 6 7 9 7 5 6 7 9 7 9 9 7 8 9 7 9 9 7 input: _GO Gilblin Stalker NAME_END 2 ATK_END 3 DEF_END 2 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END NIL RACE_END Common RARITY_END b Stealth b _EOS output: _GO _GO class MinionCard MinionCard def init self super super ground truth: _GO class GilblinStalker MinionCard def init self super init Gilblin Stalker 2 CHARACTER_CLASS ALL CARD_RARITY COMMON def create_minion self player return Minion 2 3 stealth True _EOS BLEU: 0.07805830

1 1 3 5 5 6 7 9 7 9 9 7 8 9 9 9 7 6 7 9 7 7 input: _GO Glaivezooka NAME_END 2 ATK_END 1 DEF_END 2 COST_END 2 DUR_END Weapon TYPE_END Hunter PLAYER_CLS_END NIL RACE_END Common RARITY_END b Battlecry b Give a random friendly minion 1 Attack _EOS output: _GO _GO class MinionCard MinionCard def init self super super ground truth: _GO class Glaivezooka WeaponCard def init self super init Glaivezooka 2 CHARACTER_CLASS HUNTER CARD_RARITY COMMON battlecry Battlecry Give ChangeAttack 1 MinionSelector None picker RandomPicker def create_weapon self player return Weapon 2 2 _EOS BLEU: 0.02914679

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 9 7 7 9 input: _GO Gnomeregan Infantry NAME_END 1 ATK_END 4 DEF_END 3 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END NIL RACE_END Common RARITY_END b Charge b NL b Taunt b _EOS output: _GO _GO class MinionCard MinionCard def super super super ground truth: _GO class GnomereganInfantry MinionCard def init self super init Gnomeregan Infantry 3 CHARACTER_CLASS ALL CARD_RARITY COMMON def create_minion self player return Minion 1 4 charge True taunt True _EOS BLEU: 0.05563019

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 7 9 9 9 7 7 9 input: _GO Goblin Blastmage NAME_END 5 ATK_END 4 DEF_END 4 COST_END 1 DUR_END Minion TYPE_END Mage PLAYER_CLS_END NIL RACE_END Rare RARITY_END b Battlecry b If you have a Mech deal 4 damage randomly split among all enemies _EOS output: _GO _GO class MinionCard MinionCard def super super super ground truth: _GO class GoblinBlastmage MinionCard def init self super init Goblin Blastmage 4 CHARACTER_CLASS MAGE CARD_RARITY RARE battlecry Battlecry Damage 1 CharacterSelector None EnemyPlayer RandomPicker 4 GreaterThan Count MinionSelector IsType MINION_TYPE MECH value 0 def create_minion self player return Minion 5 4 _EOS BLEU: 0.01312189

1 1 3 5 5 6 7 9 7 9 9 8 9 7 7 7 9 9 9 7 7 9 input: _GO Goblin Sapper NAME_END 2 ATK_END 4 DEF_END 3 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END NIL RACE_END Rare RARITY_END Has 4 Attack while your opponent has 6 or more cards in hand _EOS output: _GO _GO class MinionCard MinionCard def super super super ground truth: _GO class GoblinSapper MinionCard def init self super init Goblin Sapper 3 CHARACTER_CLASS ALL CARD_RARITY RARE def create_minion self player return Minion 2 4 buffs Buff ChangeAttack 4 GreaterThan Count CardSelector EnemyPlayer value 5 _EOS BLEU: 0.02856149

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 6 7 9 7 7 input: _GO Grove Tender NAME_END 2 ATK_END 4 DEF_END 3 COST_END 1 DUR_END Minion TYPE_END Druid PLAYER_CLS_END NIL RACE_END Rare RARITY_END b Choose One b Give each player a Mana Crystal or Each player draws a card _EOS output: _GO _GO class MinionCard MinionCard def init self super super ground truth: _GO class GroveTender MinionCard def init self super init Grove Tender 3 CHARACTER_CLASS DRUID CARD_RARITY RARE choices Choice GiftOfMana GiveManaCrystal PlayerSelector players BothPlayer Choice GiftOfCards Draw PlayerSelector players BothPlayer def create_minion self player return Minion 2 4 _EOS BLEU: 0.02598335

1 1 3 5 5 6 7 9 7 9 9 7 8 9 7 9 9 7 9 7 6 7 input: _GO Hemet Nesingwary NAME_END 6 ATK_END 3 DEF_END 5 COST_END 1 DUR_END Minion TYPE_END Neutral PLAYER_CLS_END NIL RACE_END Legendary RARITY_END b Battlecry b Destroy a Beast _EOS output: _GO _GO class MinionCard def init super ground truth: _GO class HemetNesingwary MinionCard def init self super init Hemet Nesingwary 5 CHARACTER_CLASS ALL CARD_RARITY LEGENDARY battlecry Battlecry Kill MinionSelector IsType MINION_TYPE BEAST players BothPlayer picker UserPicker def create_minion self player return Minion 6 3 _EOS BLEU: 0.00741695

I hope you can give me some advice about this issue. Thanks! 2.**About the best result.** I am trying to get the best result in the paper, but I seem to have some difficulty in running it. Although I have read the Parameter Settings section of the paper, I am still a bit confused as to how to get the best results. I would appreciate it if you can provide any file such as the eval.py,run.py or any other files that can be run to get the best results as a reference. Thank you for sharing your code and your hard work! It would be great if you could give me some guidance, thanks!

RuYunW / ADG-Seq2Seq

About the result of the eval.py HS, and the best results #1