Closed findmyway closed 4 years ago
Thanks, Jun, for this great summary! Here are some initial comments from my side (more to come later)
show
and use fieldnames
to see what data lies around; see also this discussion) but I think the way Flux.jl implements callbacks one could use either way. For example
rlsetup = RLSetup(learner, buffer, policy)
reward = Float64[]
callback = () -> push!(rewards, rlsetup.buffer.rewards[end])
Currently, the learner contains the parameters that are necessary for exploitation and the policy contains extra information to specify exploitation.
The function selectaction
implements the exploration-exploitation trade-off through its dependence on both the learner
and the policy
. The function update
obviously changes just the parameters. This is also the case for DeepActorCritic
, despite the confusing fieldnames policynet
and policylayer
: these are just the parameters necessary for exploitation and it is only by choosing the default policy to be a soft-max policy that exploration becomes fully specified.
Maybe it should become clearer in the naming that we are really dealing here with the exploitation-exploration trade-off, but I think that this separation needs to persist. It should also in the future be possible to run DQN with an arbitrary choice of exploration method, even though the default one may be epsilon-greedy
.
I would prefer to not store any data twice in any buffer. It is just nice to load the longest possible replay buffer fully onto a GPU for fast training; reducing the length of the longest possible replay buffer by almost a factor 2 because of redundant storage, seems not nice to me. I would prefer to have a clean and unambiguous access to the replay buffer. I think we never really discussed proposition #22. Maybe this could be a way forward.
I agree, traces are like buffers. They are used for methods with eligibility traces like TD-lambda.
Yes, we should really figure out how to design these before moving on :smile:. See also this question.
I think only a function callback!
isn't enough.
I will adopt the idea from JuliaML/LearningStrategies.jl
.
This package provides a tranning loop learn!
:
https://github.com/JuliaML/LearningStrategies.jl#metastrategy
There are five different types of callback functions for the differnet needs in different stages of trainning.
"setup!
, update!
, hook
, finished
, cleanup!
"
So, in case of RL, maybe we need to extend it for espodic task starting/ending.
Also, the tracers design is included in that package. For example, a iteration limiter https://github.com/JuliaML/LearningStrategies.jl#learning-with-a-metalearner
Or a dataframe tracer (it may be useful for tracing rewards): https://github.com/JuliaML/LearningStrategies.jl/blob/master/examples/dftracer.jl
I propose to introducing a familly of callback functions:
setup!
setup!(::EpStart, ...)
update!
hook
finished
finished(::EpEnd, ...)
cleanup!
It's okay to changing naming, I just enumerated my idea casually.
@jbrea I agree with the comments about Buffer and we can discuss details later. I think I still need a couple of days to figure out how to borrow some ideas about distributed algorithms from rllib .
@iblis17 Good suggestion! I also find that there are different callbacks in different stages in rllib.
Here I'd like to share some random thoughts on this package in the following three aspects:
Feel free to correct me if I misunderstand anything here.
What we have?
RLSetup
RLSetup
is used to organize all the necessary information in the training process. It combines agent(learner
andpolicy
here), environment and some parameters (likestoppingcriterion
,callbacks
...) together. Then we can calllearn!
for training andrun!
for testing.Comments:
RLSetup
is very common and useful in software development. (A very similar concept is TestSuite) And it makes the parameters ofcallback!
function (which I'll describe soon) consistent. Because everything we need in a callback are all wrapped in aRLSetup
! My only concern is that, different algorithms may need different kinds of parameters for (distributed) training and testing. It is a little vague to cover all these cases in a concept of RLSetup. It would be better to move the extra parameters (likestoppingcriterion
,callbacks
...) into thelearn!
andrun!
functions. And only keep the core components likelearner
,buffer
,policy
in theRLSetup
.stoppingcriterion
andcallbacks
seem to share some similarities. I tried to generalize these two here. I haven't test whether there's any performance decrease here. And by doing so it will letstoppingcriterion
to have multiple criterions.callbacks
callbacks
are useful for debugging and statistics. Currently, to define a customized callback, we need to do something like this:Comments:
callback!
function can be further simplified with a more general definitioncallback!(c, rlsetup, sraw, a, r, done) = callback!(c,rlsetup)
considering that we don't needsraw, a, r, done
in most cases.Learner and Policy
Two core functions around a learner is
selectaction
andupdate
.selectaction(learner, policy, state)
is called in each step(insidestep!
) to generate an action. (maybe call it actor would be better?)update(learner, buffer)
is called inside `learn! to update a learner.And we already have several well tested learners
Comments:
learner
is not very clear in the package (I mean it is too generic here and maybe we can decompose it into several common components?).Buffer
Here buffer is used for experience replay. One of the most useful buffer is
ArrayStateBuffer
. It uses a circular buffer to store experiences.Comments:
Traces
To be honest, I haven't look into the applications of this part. But by reading the source code, I'm wondering if it could be integrated into the concept of buffer. @jbrea
Environment
Environment related code has been split into ReinforcementLearningBase. As @jobjob suggested, we'd better to create a new repo (like Plots.jl I guess?) to support different backends. And we can have many different wrappers to easily introduce new environments. Preprocessors can also be merged into wrappers. I'll make an example repo later and have more discussions there.
Conclusion
In my humble opinion, the components listed above are clear enough to solve many typical RL problems in single machine. For continuous action space problems, @jbrea will take a look later. The only work left is to reorganize the source code a little and clearly define some abstract structs to guide developers on how to implement new algorithms. Some highlights in this repo are:
What are missing?
To compete with many other packages in RL, there's still a long way to go. And one of the most important part is to support distributed RL algorithms.
Typically, there are two directions to scale up deep reinforcement learning.
For the first one, we need an efficient parameter server and a standalone resource manager to dispatch computations. (I'm not very experienced in this field, you guys may add more details here.) Some questions in mind are:
For the seconde one, I think we need to carefully design the api first. Although there are many implementations here in Dopamine and here in ray, none of them can be directly ported into Julia (and I believe we can have more efficient implementations). Some critical points are:
Multi-agent
Although multi-agent scenarios are not considered in most existing packages, we'd better to think about it in the early stage.
Model Based Algorithms
Compared with Ray
According to the paper about Ray, there are three system layers:
For me, the first and second part is relatively easier to understand and re-implement. But the third part is especially difficult for me to figure out how to do it in Julia. If I understand it correctly, Arrow, Plasma is used for processors in one node to avoid serialization/deserialization. I've checked the package Arrow.jl, it seems there's only data transformation and I still don't know how to manage a big memory shared Object Store in Julia across processors like the one in ray.
For the rllib part, the different levels of abstractions are really worth learning from.
So for me, I'm more skilled in implementing the Env Creator part and I can also offer help to design the API of the other parts. But for the system design level, I really feel that I have a lot to learn.
What's the ideal way to do RL research in Julia?
TODO List