Using the Toolbox to solve the Tag problem

[TurquoiseKitty]

I am trying to use this toolbox to solve the tag problem mentioned in the PBVI paper, and i have the following two questions:

1. To implement a POMDP.PBVI solver, the python document shows that, when calling the solver __call__( (PBVI)self, (Model)model, (POMDP_VFun)v) -> object : which means i need to pass an addition POMDP_VFun object to make the solver work. I created an empty POMDP_VFun object and the PBVI solver do work on the simple tiger_door problem, but i still wonder what is this function for? and how to actually define a meanful POMDP_VFun object?
2. In the environment of the tag problem, according to the actual position, the robot may have restricted potential actions. For example, if the robot is on the north edge, then its next step cannot move north. How to include these restrictions into the environment of the model?

[Svalorzen]

1

I'm not sure whether you noticed that every function is documented. If in the Python command line you type

help(AIToolbox.POMDP.PBVI)

you'll get a full explanation of the class and methods. The docs are also pretty much the same as the equivalent C++ version, so feel free to go over there as well.

Defining a VFun in Python is possible, but in general these types are symmetric to the C++ ones (when possible, sometimes it is not possible to build them in Python directly, depending on what I have wrapped). It's mostly a list of lists. The documentation for the lower level stuff exists, but might be a bit more hidden. In general I would advise looking around a bit in the C++ header files (in the include folder) to orient yourself even if you do not plan to use C++.

In any case, the docs certainly could use improvement, so if you feel something is unclear and have ideas to make them better, please do open an issue and we can discuss how you can contribute :)

2

The toolbox at the moment does not include changing action spaces. This was a design decision that I made at the beginning, as with action spaces that depend on the current state every single algorithm needs to be aware of that and have code which handles lists of actions for each state.

You can emulate that by simply having the north action do something else. This might make learning/planning a bit more inefficient since you have dupes, but trust me when I say that it greatly simplifies the code of the library. Since I'm mostly maintaining this alone, I figured it would be a reasonable tradeoff.

[TurquoiseKitty]

right that make sense!

so is it appropriate that every time i use PBVI, i just pass an empty VFun to it?

and one more question is that how will the toolbox deal with it if i want the markov process to stop when hitting certain state?

[Svalorzen]

The default parameter should be empty anyway, so I think you can also not pass anything and it should work the same.

For planning algorithms, I usually just have my MDP return the same state over and over with 0 reward. For learning, you just stop the episode when you reach the terminal state.

[TurquoiseKitty]

alright! Problems solved! you are really helpful! :)