Problems of MARL - PPO:

[skydvn]

1. We have not insert the training of Critic for PPO.
2. We should split the training of Actor and Critic separately, to be more specific:
   • The Actor will use the advantage loss.
   • The Critic will use the MSE loss between infered Q from Critic + true Q