pocokhc
commented
4 years ago Thank you for your learning results. It's wonderful that you can move 16actor!
My question is, is this the right way to increase the number of actors? Another question is, how many actors can i run in my system?
Yes it is.
Although it is allocation of physical resources, it is 1actor 1CPU. So if you follow the paper, 256 CPU is required.
If you want to implement this repository, you can do it below, but I have not tested it because there is no verification environment.
class MyActor(ActorUser):
(snip)
class MyActor1(MyActor):
allocate = "/device:CPU:0"
class MyActor2(MyActor):
allocate = "/device:CPU:1"
class MyActor3(MyActor):
allocate = "/device:CPU:2"
class MyActor4(MyActor):
allocate = "/device:CPU:3"
...
Also, the mechanism that can dynamically input "allocate" with for is not implemented
Also what you think about my selection of epsilon values for every actor?
This is also proposed by Ape-X(DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY), and each actor determines epsilon by the following formula.
The code can be achieved with:
(snip)
from agent.policy import EpsilonGreedyActor
(snip)
class MyActor(ActorUser):
def getPolicy(self, actor_index, actor_num):
return EpsilonGreedyActor(actor_index, actor_num, epsilon=0.4, alpha=7)
def fit(self, index, agent):
(snip)
(snip)
kwargs["actors"] = []
for i in range(256):
kwargs["actors"].append(MyActor)
I hope it helps you.
=======================================
学習結果ありがとうございます。 16actorも動かせるなんてすばらしいです！
My question is, is this the right way to increase the number of actors? Another question is, how many actors can i run in my system?
はい、合っています。
物理リソースの割り当てですが、1actor 1CPUです。 なので論文通りの場合は256CPUが必要です。
一応本レポジトリで実現する場合は以下でできるようになっていますが検証環境がないのでテストはしていません。
class MyActor(ActorUser):
(snip)
class MyActor1(MyActor):
allocate = "/device:CPU:0"
class MyActor2(MyActor):
allocate = "/device:CPU:1"
class MyActor3(MyActor):
allocate = "/device:CPU:2"
class MyActor4(MyActor):
allocate = "/device:CPU:3"
...また、allocate は for 等で動的に入力できるようには実装していません。。。
Also what you think about my selection of epsilon values for every actor?
こちらも Ape-X で提案されており、各Actorは以下の式でepsilonを決めています。
コードでは以下のように実装できます。
(snip)
from agent.policy import EpsilonGreedyActor
(snip)
class MyActor(ActorUser):
def getPolicy(self, actor_index, actor_num):
return EpsilonGreedyActor(actor_index, actor_num, epsilon=0.4, alpha=7)
def fit(self, index, agent):
(snip)
(snip)
kwargs["actors"] = []
for i in range(512):
kwargs["actors"].append(MyActor)
Gaethje
I am trying to run Alien-v0 with 16 actors. My code is like this-
I have run this on my machine and after running around 4 hours, the maximum reward is 970. My question is, is this the right way to increase the number of actors? Another question is, how many actors can i run in my system? how to calculate the number of actors? For example- in the paper they used 256 actors. How much physical resource do i need to work with 256 actors? Also what you think about my selection of epsilon values for every actor? it is referenced that the epsilon is chosen according the formula of this paper- DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY formula they mentioned for choosing epsilon for every actor-This is mentioned in section 4.1-atari.
Great repository! thanks!