The score of `train_dqn_gym.py` with `--actor-learner` is lower than the baseline score.

[shu65]

It seems that the score of train_dqn_gym.py with --actor-learner is lower than the baseline score.

the result of train_dqn_gym.py with --actor-learner

$python train_dqn_gym.py --env CartPole-v0 --gpu -1 --actor-learner

$cat scores.txt
steps   episodes        elapsed mean    median  stdev   max     min     average_q       average_loss    cumulative_steps        n_updates       rlen
10102   295     27.7667019367218        142.13  143.5   31.933850505080155      200.0   66.0    1.1826544       0.002722540542599745    10102   3161    10102
20125   377     50.456093072891235      153.35  153.5   28.010595037296767      200.0   89.0    0.9678241       0.0008152683400840032   20125   6378    20125
30044   433     73.47023296356201       183.16  200.0   29.932152234240313      200.0   89.0    0.81571275      0.001155181206850102    30044   10039   30044
40125   504     95.93452501296997       166.15  187.5   42.48193317711244       200.0   21.0    0.81241447      0.0008331169206940104   40125   13427   40125
50072   567     118.65022873878479      179.05  200.0   31.635989925808527      200.0   105.0   0.76570725      0.0011051068906817818   50072   16768   50072
60069   630     141.68070459365845      140.9   119.0   38.606143856054096      200.0   99.0    0.7382145       0.0004422377867740579   60069   19830   60069
70140   688     165.85664129257202      195.39  200.0   13.644768117109832      200.0   135.0   0.6611406       0.00039179762254207165  70140   23619   70140
80124   754     190.2280843257904       187.3   200.0   20.73424922463051       200.0   125.0   0.56931555      0.00041157132740408996  80124   27049   80124
90022   807     216.45293831825256      188.56  200.0   26.396479716320915      200.0   99.0    0.47856167      0.0001967151611461304   90022   30542   90022
100000  866     244.79555416107178      188.37  200.0   21.623806091874528      200.0   110.0   0.3955511       0.00030046377703911274  100000  34203   100000

the result of the baseline (without --actor-learner)

$python train_dqn_gym.py --env CartPole-v0 --gpu -1

$cat scores.txt
steps   episodes        elapsed mean    median  stdev   max     min     average_q       average_loss    cumulative_steps        n_updates       rlen
10036   229     55.154892921447754      123.64  121.0   9.809828095982482       161.0   111.0   2.9230406       0.012885378097416833    10036   9037    10036
20026   320     107.90914940834045      98.46   99.0    3.4035631952767416      108.0   91.0    4.9529943       0.024560987005243076    20026   19027   20026
30177   400     164.05463528633118      197.4   199.0   2.9059326290271157      200.0   190.0   4.6993313       0.0120574060222134      30177   29178   30177
40074   458     215.77593541145325      200.0   200.0   0.0     200.0   200.0   4.9031615       0.021290321972919628    40074   39075   40074
50089   520     270.9923541545868       180.73  200.0   29.913631567152684      200.0   123.0   4.1760216       0.009463735535391607    50089   49090   50089
60020   596     328.6313564777374       200.0   200.0   0.0     200.0   200.0   3.2222836       0.00954763395129703     60020   59021   60020
70101   649     388.06564927101135      200.0   200.0   0.0     200.0   200.0   2.484032        0.006800949496391695    70101   69102   70101
80088   707     444.6856653690338       200.0   200.0   0.0     200.0   200.0   1.7810422       0.00335951144239516     80088   79089   80088
90091   768     501.1645920276642       200.0   200.0   0.0     200.0   200.0   1.1870649       0.0015449915380304447   90091   89092   90091
100000  830     557.4196665287018       125.91  126.0   3.8193949491742507      133.0   117.0   0.78206486      0.0015336718078833656   100000  99001   100000

[muupan]

Possible hypothesis:

• Just variance. At least both seems successful, reaching median score of 200.
• Difference in n_updates. Actor-learner can collect data faster, but model updates are not always faster. After 100k steps, n_updates is 34203 with --actor-learner and 99001 without --actor-learner, thus it makes sense the latter performs better.

Thus, I'm not sure if it is an issue that needs a fix.