Fancy declarative syntax for a subset of problems

[zsunberg]

In talking with @odow at Juliacon, I started thinking that POMDPs.jl is still quite difficult to use because much knowledge must be assimilated before even getting started, and much functionality (notably discretization) is still missing. This is especially true for people new to julia or engineers who are not that experienced at programming. Many people who start using this software don't even really know what an interface is and, even if they do, Julian interfaces look much different than they do in other languages, so, before they can use the package, they must learn about Julian interfaces.

One partial solution to this would be to create a JuMP-esque declarative syntax for specifying a subset of problems (e.g. maybe we could even handle all MDPs?). It might look something like this https://github.com/odow/SDDP.jl/blob/master/examples/FAST/hydro_thermal.jl . For example, there would be a @state macro to declare the state space, and @action macro for the action space, etc.

The current POMDPs.jl interface could function as a common backend for solvers and problems as MathProgBase currently does.

[zsunberg]

I unfortunately do not have the time to implement this - just throwing it out there.

[zsunberg]

I think that @haoyio tried something like this a while back with PLite. I think it might come out a lot better now that the language is more mature and we have more examples like JuMP to see how metaprogramming can be used well.

[odow]

I think a better example is this one https://github.com/odow/DynamicProgramming.jl/blob/master/examples/battery_storage.jl

It solves using backwards recursion.

Bonus points

• [ ] Find an approach that unifies MDP's and Stochastic Programming mindsets
• [ ] Allow my dynamic programming examples to hook into the POMDPs value iteration solution algorithm
• [ ] Identify when the value function of an MDP is convex w.r.t. the state space, and the states are continuous and everything is linear/quadratic and solve via SDDP.jl

[zsunberg]

@odow thanks for the suggestions - yeah, it would be good for someone familiar with the stochastic programming literature and the MDP literature to write down a comparison that matches terms, identifies differences, etc.

[odow]

http://castlelab.princeton.edu/cso.htm

http://castlelab.princeton.edu/Papers/Powell-UnifiedFrameworkforStochasticOptimization_April162016.pdf

[odow]

cc @blegat @leclere @frapac

[odow]

cc'ing @pierre-haessig who has a python library for stochastic control with some interesting ideas https://github.com/pierre-haessig/stodynprog/blob/master/examples/howto%20storage-AR1.ipynb and just posted an issue in JuMP (so is presumably interested in Julia)

[zsunberg]

@odow, thanks for the links - this definitely gives me a better perspective for the breadth of the research that is out there.

[zsunberg]

I created this repo to work on this: https://github.com/JuliaPOMDP/QuickPOMDPs.jl

[pierre-haessig]

Thanks for putting me in the loop. I'm not developping much dynamic programming code these days, but I'm glad to see neat syntax proposals in this new QuickPOMDP.