search

ShangtongZhang / DeepRL

Modularized Implementation of Deep RL Algorithms in PyTorch
MIT License
3.21k stars 686 forks source link

Just a quick question #27

Closed hohoCode closed 6 years ago

hohoCode commented 6 years ago

https://github.com/ShangtongZhang/DeepRL/blob/e427e8f73f7d6c6ae0283a7a4438b724725ec192/agent/QuantileRegressionDQN_agent.py#L67-L68

Just a few quick questions...:

  1. The '* self.quantile_weight' component seems to be not really relevant here because doing the multiplication will not change the relative order of each action value, therefore no matter it is multiplied or not, all action values stay the same order.

  2. Since quantiles_next has dim=-1 softmaxed (right?), therefore all the actions in this line will have the same values. Just wondering if this is correct.

Thanks for your codes.

ShangtongZhang commented 6 years ago
  1. yes, just want to compute the real q value
  2. quantiles_next is not softmaxed, the one that is softmaxed is categorical DQN.
hohoCode commented 6 years ago

Thanks for the clarifications.