Open slwu89 opened 9 months ago
Hi @slwu89, sorry for the late reply. This looks fun! I have yet to neaten up the environments within the repository; just use Pkg.activate/Pkg.add/Pkg.instantiate to use a custom environment for the tutorial, and then everything should work fine.
The output doesn't look too promising though! This framework may be better suited to the 'flattening the curve' scenario, as I've implemented (using JuMP) here.
Thanks @sdwfrost, will do that.
Yeah, it does look distressingly similar to the SDDP example. I guess not too surprising, as they are both alternative ways to solve approximations of the Bellman equation for forward look ahead type problems. I'll check how it works with that alternative scenario. Perhaps I'll also modify SDDP to work on that example too.
[EDIT]: actually I think MDP and SDDP are perhaps not well suited to the "flattening the curve" scenario, at the stochastic transitions mean that constraints such as @constraint(model, [t=2:(T+1)], 0 ≤ I[t] ≤ I_max)
cannot be fulfilled. If it makes sense to do so, "goal programming" could be used such that deviations from that constraint could be minimized in the objective (SDDP) or penalized in the reward function (MDP).
You're right that 'soft' constraints may allow MDP/SDDP to work better in the flattening the curve context. If you get POMDP.jl to work in this setting, it would also be interesting to use something like https://juliareinforcementlearning.org (which also implements a MDP).
I'll give those soft constraints as penalties a try when I get some time for them! Thanks for the link, I haven't seen that before.
Just pinging @slwu89 to see if I can help unblock things...
Not enough time lately, hopefully soon!
How soon is now? :)
Haha, good question. The issue is that the soft constraints and goal programming formulation would need more time/effort beyond than simple coding; I'm not sure what's the best expression to enforce such penalties on the optimizer.
Hi @sdwfrost, I'm opening this as a draft because it is clearly a work in progress, but also because I've had some problems with package satisfiability requirements and I wonder if you could help check it out, as I'm not sure from which tutorial the offending package is coming from.
I'm on Julia 1.9.3 and when I attempt to run
add POMDPs, POMDPTools, QuickPOMDPs, MCTS
, I get the following error: