hongzimao
commented
4 years ago Can you sanity check A2C improves from a random policy after 1e8 time steps? If the policy was improving, we need to tune the training process; if not, the A2C implementation itself might have some problem.
You can try a few techniques like curriculum learning (short episode in the beginning of training then gradually increase the episode length to make the learning task more and more challenging, similar to section 5.3 challenge 1 of this paper http://people.csail.mit.edu/hongzi/content/publications/Decima-Sigcomm19.pdf). You can also implement an input-dependent baseline (https://openreview.net/forum?id=Hyg1G2AqtQ). For this problem, you can do a time-dependent baseline for simplicity: just make the parallel agent experience the same input sequence (i.e., job sequence) and compute the baseline return on each time step by averaging on the same time step across these parallel agents.
RuihangWang
Hi, Hongzi.
I am trying to reproduce the results of load balancer using A2C from stable-baselines. I followed the experimental settings described in the ICML paper, and also clipped the gradients by setting the max_grad_norm to 10. However, the performance of the trained A2C agent cannot beat the baseline greedy policy after 1e8 time steps.
The greedy policy achieves episode reward -7e5 while the A2C agent only achieves -2.74e6 for the best test. It seems the greedy policy is a better strategy compared with DRL.
Maybe the default settings of other hyperparameters are not suitable. I am wondering if there are other tricks when training the load balancer to achieve satisfied performance. Looking forward to your reply. Thanks!