search

saizhang0218 / TMC

Pytorch implementation of "Succinct and Robust Multi-Agent Communication With Temporal Message Control"
26 stars 10 forks source link

The experimental results cannot be reproduced on 6h_vs_8z #3

Closed TianQi-777 closed 1 year ago

TianQi-777 commented 3 years ago

I tried to reproduce the performance of TMC on 6h_vs_8z, but all failed (regardless of SC2.4.6 (the paper default) or SC2.4.10 version), I ran the following command

src/main.py --config=qmix_6h_vs_8z --env-config=sc2 with env_args.map_name=6h_vs_8z

The results is as follows:

{   'action_selector': 'epsilon_greedy',
    'agent': 'rnn',
    'agent_action_decoder': 'rnn_action_decoder',
    'agent_output_type': 'q',
    'batch_size': 32,
    'batch_size_run': 8,
    'buffer_cpu_only': True,
    'buffer_size': 5000,
    'checkpoint_path': '',
    'critic_lr': 0.0005,
    'delta': 0.05,
    'double_q': True,
    'env': 'sc2',
    'env_args': {   'continuing_episode': False,
                    'debug': False,
                    'difficulty': '7',
                    'game_version': None,
                    'heuristic_ai': False,
                    'map_name': '6h_vs_8z',
                    'move_amount': 2,
                    'obs_all_health': True,
                    'obs_instead_of_state': False,
                    'obs_last_action': False,
                    'obs_own_health': True,
                    'obs_pathing_grid': False,
                    'obs_terrain_height': False,
                    'obs_timestep_number': False,
                    'replay_dir': '',
                    'replay_prefix': '',
                    'reward_death_value': 10,
                    'reward_defeat': 0,
                    'reward_negative_scale': 0.5,
                    'reward_only_positive': True,
                    'reward_scale': True,
                    'reward_scale_rate': 20,
                    'reward_sparse': False,
                    'reward_win': 200,
                    'seed': 704265453,
                    'state_last_action': True,
                    'state_timestep_number': False,
                    'step_mul': 8},
    'epsilon_anneal_time': 50000,
    'epsilon_finish': 0.05,
    'epsilon_start': 1.0,
    'evaluate': False,
    'fresh_limit': 6,
    'gamma': 0.99,
    'grad_norm_clip': 10,
    'label': 'default_label',
    'learner': 'q_learner_6h_vs_8z',
    'learner_log_interval': 20000,
    'load_step': 0,
    'local_results_path': 'result_3s_vs_4z',
    'log_interval': 20000,
    'loss_level': 'none',
    'lr': 0.0005,
    'mac': 'basic_mac_6h_vs_8z',
    'mixer': 'qmix',
    'mixing_embed_dim': 32,
    'name': 'vdn_smac_parallel',
    'obs_agent_id': True,
    'obs_last_action': True,
    'optim_alpha': 0.99,
    'optim_eps': 1e-05,
    'repeat_id': 1,
    'rnn_hidden_action_decoder_dim': 16,
    'rnn_hidden_dim': 64,
    'runner': 'parallel',
    'runner_log_interval': 20000,
    'save_model': True,
    'save_model_interval': 2000000,
    'save_replay': False,
    'seed': 704265453,
    't_max': 10050000,
    'target_update_interval': 200,
    'test_greedy': True,
    'test_interval': 20000,
    'test_nepisode': 24,
    'transmit_limit': 6,
    'use_cuda': True,
    'use_tensorboard': False}
Using default stable ids, none found at: /home/tianqi/TMC/3rdparty/StarCraftII/stableid.json
Successfully loaded stable ids: GameData\stableid.json
Sending ResponseJoinGame
[INFO 14:33:03] my_main t_env: 195 / 10050000
[INFO 14:33:03] my_main Estimated time left: 1 hours, 6 minutes, 14 seconds. Time passed: 7 seconds
[INFO 14:33:05] my_main Saving models to result_3s_vs_4z/models/vdn_smac_parallel__2021-05-10_14-32-52/195
[INFO 14:33:25] my_main Updated target network
[INFO 14:33:45] my_main Updated target network
[INFO 14:34:05] my_main Updated target network
[INFO 14:34:25] my_main Updated target network
[INFO 14:34:31] my_main Recent Stats | t_env:      20057 | Episode:      872
battle_won_mean:           0.0000   dead_allies_mean:          6.0000   dead_enemies_mean:         0.0000   ep_length_mean:           24.3750
epsilon:                   1.0000   grad_norm:                23.8551   loss:                      4.4272   q_taken_mean:              0.0465
return_mean:               5.1924   return_std:                0.5085   target_mean:               0.3280   td_error_abs:              1.6974
test_battle_won_mean:      0.0000   test_dead_allies_mean:     6.0000   test_dead_enemies_mean:    0.5000   test_ep_length_mean:      23.7500
test_return_mean:          2.1548   test_return_std:           0.4908   
[INFO 14:34:32] my_main t_env: 20235 / 10050000
[INFO 14:34:32] my_main Estimated time left: 12 hours, 27 minutes, 39 seconds. Time passed: 1 minutes, 37 seconds
[INFO 14:34:47] my_main Updated target network
[INFO 14:35:07] my_main Updated target network
[INFO 14:35:29] my_main Updated target network
[INFO 14:35:52] my_main Updated target network
[INFO 14:36:11] my_main Recent Stats | t_env:      40072 | Episode:     1760
battle_won_mean:           0.0000   dead_allies_mean:          6.0000   dead_enemies_mean:         0.0499   ep_length_mean:           23.6783
epsilon:                   0.6189   grad_norm:                12.0444   loss:                      2.2370   q_taken_mean:              0.1825
return_mean:               5.4840   return_std:                0.5408   target_mean:               0.3222   td_error_abs:              0.9322
test_battle_won_mean:      0.0000   test_dead_allies_mean:     6.0000   test_dead_enemies_mean:    1.2083   test_ep_length_mean:      23.1667
test_return_mean:          4.8228   test_return_std:           0.4864   
[INFO 14:36:12] my_main t_env: 40250 / 10050000
[INFO 14:36:12] my_main Estimated time left: 13 hours, 49 minutes, 22 seconds. Time passed: 3 minutes, 17 seconds
[INFO 14:36:19] my_main Updated target network
[INFO 14:36:43] my_main Updated target network
[INFO 14:37:07] my_main Updated target network
[INFO 14:37:30] my_main Updated target network
[INFO 14:37:54] my_main Updated target network
[INFO 14:38:00] my_main Recent Stats | t_env:      60155 | Episode:     2656
battle_won_mean:           0.0000   dead_allies_mean:          6.0000   dead_enemies_mean:         0.1819   ep_length_mean:           23.2987
epsilon:                   0.2386   grad_norm:                 8.2995   loss:                      1.5073   q_taken_mean:              0.2746
return_mean:               5.8364   return_std:                0.5519   target_mean:               0.3699   td_error_abs:              0.6779
test_battle_won_mean:      0.0000   test_dead_allies_mean:     6.0000   test_dead_enemies_mean:    1.1944   test_ep_length_mean:      22.6806
test_return_mean:          5.6935   test_return_std:           0.5650   

...

[INFO 09:38:59] my_main t_env: 8443380 / 10050000
[INFO 09:38:59] my_main Estimated time left: 3 hours, 52 minutes, 47 seconds. Time passed: 19 hours, 6 minutes, 4 seconds
[INFO 09:39:05] my_main Recent Stats | t_env:    8443380 | Episode:   331792
battle_won_mean:           0.0025   dead_allies_mean:          5.9931   dead_enemies_mean:         1.5178   ep_length_mean:           27.9811
epsilon:                   0.0500   grad_norm:                 1.2608   loss:                      0.1001   q_taken_mean:              0.8819
return_mean:              10.8696   return_std:                1.6117   target_mean:               0.8738   td_error_abs:              0.2224
test_battle_won_mean:      0.0000   test_dead_allies_mean:     6.0000   test_dead_enemies_mean:    1.8500   test_ep_length_mean:      28.4917
test_return_mean:         11.2949   test_return_std:           1.4977   
[INFO 09:39:09] my_main Updated target network
[INFO 09:39:53] my_main Updated target network
[INFO 09:40:37] my_main Updated target network
[INFO 09:41:24] my_main Updated target network
[INFO 09:41:46] my_main t_env: 8463541 / 10050000
[INFO 09:41:46] my_main Estimated time left: 3 hours, 39 minutes, 16 seconds. Time passed: 19 hours, 8 minutes, 51 seconds
[INFO 09:41:53] my_main Recent Stats | t_env:    8463541 | Episode:   332504
battle_won_mean:           0.0039   dead_allies_mean:          5.9886   dead_enemies_mean:         1.5494   ep_length_mean:           28.0710
epsilon:                   0.0500   grad_norm:                 1.6167   loss:                      0.1016   q_taken_mean:              0.8891
return_mean:              10.9093   return_std:                1.6683   target_mean:               0.8777   td_error_abs:              0.2260
test_battle_won_mean:      0.0083   test_dead_allies_mean:     5.9917   test_dead_enemies_mean:    1.9333   test_ep_length_mean:      28.8167
test_return_mean:         11.4015   test_return_std:           1.6402   
[INFO 09:42:18] my_main Updated target network
[INFO 09:43:05] my_main Updated target network
[INFO 09:43:53] my_main Updated target network
[INFO 09:44:39] my_main Updated target network
[INFO 09:44:43] my_main t_env: 8483617 / 10050000
[INFO 09:44:43] my_main Estimated time left: 3 hours, 50 minutes, 9 seconds. Time passed: 19 hours, 11 minutes, 48 seconds
[INFO 09:44:49] my_main Recent Stats | t_env:    8483617 | Episode:   333224
battle_won_mean:           0.0042   dead_allies_mean:          5.9875   dead_enemies_mean:         1.5514   ep_length_mean:           28.0579
epsilon:                   0.0500   grad_norm:                 1.3505   loss:                      0.0986   q_taken_mean:              0.8740
return_mean:              10.9086   return_std:                1.6640   target_mean:               0.8645   td_error_abs:              0.2238
test_battle_won_mean:      0.0083   test_dead_allies_mean:     5.9917   test_dead_enemies_mean:    1.9167   test_ep_length_mean:      28.9083
test_return_mean:         11.4437   test_return_std:           1.6465   
[INFO 09:45:32] my_main Updated target network
[INFO 09:46:22] my_main Updated target network
[INFO 09:47:10] my_main Updated target network
[INFO 09:47:41] my_main t_env: 8503672 / 10050000
[INFO 09:47:41] my_main Estimated time left: 3 hours, 48 minutes, 22 seconds. Time passed: 19 hours, 14 minutes, 46 seconds
[INFO 09:47:47] my_main Recent Stats | t_env:    8503672 | Episode:   333936
battle_won_mean:           0.0045   dead_allies_mean:          5.9852   dead_enemies_mean:         1.5484   ep_length_mean:           28.0783
epsilon:                   0.0500   grad_norm:                 1.4670   loss:                      0.0988   q_taken_mean:              0.8631
return_mean:              10.9099   return_std:                1.6836   target_mean:               0.8545   td_error_abs:              0.2245
test_battle_won_mean:      0.0083   test_dead_allies_mean:     5.9917   test_dead_enemies_mean:    1.8583   test_ep_length_mean:      28.9333
test_return_mean:         11.4701   test_return_std:           1.5476   
[INFO 09:48:04] my_main Updated target network
[INFO 09:48:53] my_main Updated target network
[INFO 09:49:42] my_main Updated target network
[INFO 09:50:30] my_main Updated target network
[INFO 09:50:40] my_main t_env: 8523681 / 10050000
[INFO 09:50:40] my_main Estimated time left: 3 hours, 47 minutes, 23 seconds. Time passed: 19 hours, 17 minutes, 45 seconds
[INFO 09:50:46] my_main Recent Stats | t_env:    8523681 | Episode:   334648
battle_won_mean:           0.0050   dead_allies_mean:          5.9837   dead_enemies_mean:         1.5750   ep_length_mean:           28.0844
epsilon:                   0.0500   grad_norm:                 1.7216   loss:                      0.1001   q_taken_mean:              0.8655
return_mean:              10.9119   return_std:                1.6954   target_mean:               0.8621   td_error_abs:              0.2268
test_battle_won_mean:      0.0083   test_dead_allies_mean:     5.9917   test_dead_enemies_mean:    1.8417   test_ep_length_mean:      28.9250
test_return_mean:         11.4559   test_return_std:           1.6604   
[INFO 09:51:24] my_main Updated target network
[INFO 09:52:13] my_main Updated target network
[INFO 09:53:02] my_main Updated target network
[INFO 09:53:38] my_main t_env: 8543705 / 10050000
[INFO 09:53:38] my_main Estimated time left: 3 hours, 43 minutes, 0 seconds. Time passed: 19 hours, 20 minutes, 43 seconds
[INFO 09:53:45] my_main Recent Stats | t_env:    8543705 | Episode:   335360
battle_won_mean:           0.0053   dead_allies_mean:          5.9834   dead_enemies_mean:         1.5888   ep_length_mean:           28.1185
epsilon:                   0.0500   grad_norm:                 1.6960   loss:                      0.0986   q_taken_mean:              0.8641
return_mean:              10.9297   return_std:                1.6945   target_mean:               0.8617   td_error_abs:              0.2248
test_battle_won_mean:      0.0083   test_dead_allies_mean:     5.9917   test_dead_enemies_mean:    1.9167   test_ep_length_mean:      29.2750
test_return_mean:         11.6199   test_return_std:           1.7489

My question is that --config=qmix_6h_vs_8z in the open source code is the config setting of TMC? Any suggestions？ Look forward to your replay.

smallQ-Q commented 3 years ago

So, my brother. Do you find the problem why can't reproduct this work? I also meet this problem and I suspect the reliability of this work.

happyfir commented 3 years ago

So, my brother. Do you find the problem why can't reproduct this work? I also meet this problem and I suspect the reliability of this work.

l also want to learn about it ,but l meet some diffcult when deploy it could you help me，and l have submit my problem at #6