search

larsys / markov_decision_making

Markov Decision Making
23 stars 3 forks source link

Usage Documentation #1

Open TarekTaha opened 10 years ago

TarekTaha commented 10 years ago

Hi,

I came across your mdm library and I was wondering if it's generic and can be used with any madp generated policies and .dpomdp models ? If so, can you please let me know how it can be used ?

Regards, Tarek

jmisrbackup commented 10 years ago

Hi Tarek,

MDM uses MADP directly to load models and policies, and so it can potentially be used to deploy control policies from any of the decision-theoretic model classes that can be represented in MADP to actual robot teams. It doesn't already support all of those models, however. Since there are many such classes (potentially everything ranging from Dec-POMDPs to MDPs), and many researchers focus on particular or non-standard variants of these models with different assumptions and requirements at run-time, MDM is designed to be easily extended by researchers for their particular purposes and applications, rather than trying to implement all of them out-of-the-box.

The main use case of MDM follows the "fork & pull" methodology: 1) Fork the repo; 2) Using MDM built-in functionalities, define how the sensor data of your robots maps to observations (or states) and how your actions map to robot controls; 3) If your model type isn't already supported, write your own controller by extending the existing MDM controller classes, and using the MADP API, to map from incoming observation (or state) data to actions. 4) (Optionally) If you're happy with your controller, you can then send us a pull request so that other researchers can readily use your models in the future.

So, while MDM already supports discrete centralized MDPs and POMDPs, if you want to implement a Dec-POMDP, for instance, you could write a Dec-POMDP "control layer" class based on the existing POMDP classes that would map observation sequences to actions based on a MADP policy.

Right now, the most complete technical description of MDM can be found in Section 7.3 of my PhD thesis here: http://users.isr.ist.utl.pt/~jmessias/content/PhDthesis_jmessias.pdf . There are also some implementation examples in Annex B. You can check the APIs online at: http://docs.ros.org/hydro/api/mdm_library/html/annotated.html ; http://docs.ros.org/hydro/api/predicate_manager/html/annotated.html; http://docs.ros.org/hydro/api/topological_tools/html/annotated.html.

Since this is a very specialized library and our user base is composed mostly of researchers, we tend to support its application closely (at least for now). So if you need any help with it just let us know. The mdm_example package is a good starting point, that can be run straight out-of-the-box (use the ISR_demo.launch file) and contains most of the ingredients already in place.

Cheers, João

TarekTaha commented 10 years ago

Dear João,

Thank you very much for your prompt and comprehensive reply. I came across your thesis few weeks ago and I was in the process of reading it, very good work there.

Currently our focus is on Dec-POMDP using MADP library, at this stage I can't ask you any meaningful questions before reading the material you recommened, forking your code and trying to implement the controller as you suggested. I think you did some good work creating this library and I am keen on using it and extending it if necessary and not re-invent the wheel. I will spend the next week or so testing your library and I will ask you some questions when I am stuck.

Cheers, Tarek