Customizing the DDN for MOMDPs : Attempt

[afansi]

Hello Guys,

I am asking what customization need to be performed on the DNNStructure of the DBNs to allow modeling MOMDPs.

I know that the DBN should be like this one

and I wondering if we could have a skeleton like this (my julia skills are very poor):

function momdp_ddn()
    DDNStructure((
            x = InputDDNNode(),
            y = InputDDNNode(),
            s = (x, y),
            a = InputDDNNode(),
            sp = DistributionDDNNode(transition),
            o = DistributionDDNNode(observation),
            r = FunctionDDNNode(reward),
            xp = sp[1],
            yp = sp[2],
           ),
           (x = (),
            y = (),
            a = (),
            sp = map(DDNNode, (:s, :a)),
            o = map(DDNNode, (:s, :a, :sp)),
            r = map(DDNNode, (:s, :a, :sp, :o)),
           )
          )
end

or

function momdp_ddn()
    DDNStructure((
            x = InputDDNNode(),
            y = InputDDNNode(),
            s = FunctionDDNNode(merge_state),
            a = InputDDNNode(),
            sp = DistributionDDNNode(transition),
            o = DistributionDDNNode(observation),
            r = FunctionDDNNode(reward),
            xp = FunctionDDNNode(split_state_hidden),
            yp = FunctionDDNNode(split_state_observable),
           ),
           (x = (),
            y = (),
            a = (),
            s = map(DDNNode, (:x, :y)),
            sp = map(DDNNode, (:s, :a)),
            xp = map(DDNNode, (:sp)),
            yp = map(DDNNode, (:sp)),
            o = map(DDNNode, (:s, :a, :sp)),
            r = map(DDNNode, (:s, :a, :sp, :o)),
           )
          )
end

Does this definition be compatible with the JuliaPOMDPs Solvers? What is a good way to deal with MOMDPs?

[zsunberg]

Hi @afansi, this is a good question, and I don't have a concise answer yet, but I am excited to start thinking about it! Both of the things that you wrote above make sense.

However, currently no POMDPs.jl solvers leverage MOMDP structure. If you just want to use the current POMDPs.jl solvers with one problem that happens to be a MOMDP, then I would recommend not messing with the DDN structure and instead just using x and y as local variables in your transition and observation code. POMDPs.jl currently makes the assumption that the only input nodes are :s and :a, so the DDNs you have above will not work with current code.

The rest of this post will be about how we would actually start supporting MOMDP solvers. If you meant to start this discussion, great! We are excited to start working on it :)

Working towards MOMDP solvers

The current DDN functionality was designed with the constrained (PO)MDP use case in mind, which is considerably simpler (just adding a :c node) than MOMDPs, so there may need to be some small additions/design changes in POMDPs.jl to get it right.

Things to keep in mind

1. A MOMDP should look like a POMDP from the outside so that vanilla POMDP solvers can work without knowing anything about MOMDPs
2. It should be straightforward to define a MOMDP explicitly or with a generative model

Initial thoughts

Given the above criteria, here are my initial thoughts:

1. We probably need to introduce a new concept, something like a DDNGroup or AliasNode or StateNode to represent s and sp
2. We should find a simple example MOMDP and write out what we want a definition to look like
3. We should look at the MOMDP support in the POMDPX file specification and support at least most of the features present there
4. We may need to introduce new syntax/functionality to handle explicit distributions for arbitrary nodes rather than just transition for :sp and observation for :o
5. We should think about what would be needed to implement a particle filter for a MOMDP. Since we will know :y exactly, the only thing that should be in each particle is :x. (@lassepe has a lot of experience with this)

More thoughts to come later!

[zsunberg]

I think the lowest-risk way to approach this is to develop all the components in the MOMDPs package and then move them here if we find that they work well. Moving to this issue: https://github.com/JuliaPOMDP/MOMDPs.jl/issues/1