consider explicit representation of Model state/action spaces

[sbenthall]

@llorracc has recommended looking into bellman, a toolkit for model-based reinforcement learning (MBRL), as inspiration for HARK.

What is bellman? It's an implementation of a kind of reinforcement learning algorithm that fits well with OpenAI's Gym library, a widely used framework for testing RL algorithms.

How might we start using these tools in HARK?

Macro problems are indeed special cases of MDP problems, though the solutions used in HARK are not currently based on RL.

We might ask: does Open AI's Gym contain any tools that can help us represent problems, in a way that is agnostic to the kind of solution?

One promising tool might be Gym's implementation of Space, which is a superclass for the domain of state and control variables. It includes support for both discrete and continuous space.

https://github.com/openai/gym/tree/master/gym/spaces

[llorracc]

@sbenthall is right in guessing that I'm interested in bellman as providing a systematic and well thought out structure for representing and codifying our problems. It has an explicit structure for each of the elements of a bellman equation (transition equations, states, controls, reward functions, etc), which is something we are sorely lacking.

The relationship between bellman and Gym is something I have not had time to look into. I take it that Gym defines a set of particular spaces.

I'm a bit puzzled by whether Gym allows continuous states and controls; the discussion at the link below suggests that controls, at least, must be discretized:

https://datascience.stackexchange.com/questions/72515/how-to-define-discrete-action-space-with-continuous-values-in-openai-gym

[sbenthall]

This is an example of a Gym environment that uses a continuous control variable:

https://gym.openai.com/envs/MountainCarContinuous-v0/

https://github.com/openai/gym/blob/master/gym/envs/classic_control/continuous_mountain_car.py#L99-L101

I have been looking at bellman and trying to see if it can do the modeling of bellman equations as is done in macroeconomics. My sense is that the library operates at the MDP level, and that the 'transition equations' used are generally not granular 'frame' transitions, but rather are a trained neural network representing the dynamics of the problem. My sense is that it would be an interesting stretch from the methods/conceptualization of the problem in HARK, but maybe not capable of feature parity with the current methods.