Overhaul: unified Markov framework

[mnwhite]

The "Markov model" overwrites nearly everything in ConsIndShock, and doesn't play nicely with other extensions. For my own personal work, I'm adding Markov states to ConsMedShockModel, and I will shortly be doing the same for ConsAggShockModel. It would be really great if we had a unified structure for the Markov solver so that it could be added to (m)any other solver(s) with minimal extra work.

The general setup is that the input markov_vars will be used to name the solver inputs that vary by Markov state. We might need to limit the framework to endogenous-grid-style methods, so that there is commonality across solution methods. It can't just loop over each Markov state and solve them independently because some of the inputs are dependent on next period's state, not this period. Might need to specify markov_next_vars and markov_curr_vars?

[sbenthall]

I'm interested in this. Has there been any progress on it since the issue was made?

[mnwhite]

No, and I think it's been (effectively) tagged as something for a major revision / rewrite of HARK. That is, I think it needs a lot of up front design work, and the programming work will accompany a wholesale replacement of almost all of the solution code, with a lot of but not necessarily the "structure" code.

On Sat, Oct 19, 2019 at 5:47 PM Sebastian Benthall notifications@github.com wrote:

I'm interested in this. Has there been any progress on it since the issue was made?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/96?email_source=notifications&email_token=ADKRAFLS6VXNFS6XM5DECBLQPN57FA5CNFSM4C33HN42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBX5KKA#issuecomment-544199976, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKRAFLAOEHUPD7DJBXJ3LTQPN57FANCNFSM4C33HN4Q .

[sbenthall]

My understanding is that a lot of the structure code, and maybe some of the solution code, is going to soon be replaced by dolark.

https://github.com/EconForge/dolark

Pablo was saying the other day that dolark may be more general than a Markov model because it can do models with continuous time, not just discrete time.

That makes my head hurt a little and it's possible that using the discrete-time Markov subset of that functionality would suffice for a first shot at a HARK overhaul.

I'm still trying to understand the formal constraints of models in HARK. I'm trying to translate from the economics modeling used here into frameworks I'm more familiar with, such as POMDP: https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process

My best understanding so far is:

• Agents have state in discrete time.
• At each time step, they observe their state and choose an action
• these actions will change their state in the next step and accrue utility.

In the more complex Market example...

• All actor actions and states also aggregate into a Market state at the end of each time step.
• The Market state updates according to rules
• The Market state propagates to each Agent at the beginning of each time step.

If I have the generality of the problem right it will be a lot easier work on clean structure code with dolark. I think it may be within the scope of this ticket.

[llorracc]

Sebastian,

I've been very interested in your efforts to grok the nature of the problems we are working on, in part because of what it teaches me about what it tells me about the starting point of very smart people whose training is in other areas.

The most fundamental issue is the extent to which, in other fields, the key element that is missing (but is present -- perhaps too much so -- in economic models) is the role of expectations.

The canonical example is stock prices. In economics or finance, you can't publish a model in which agents in the model would be able to predict, with a high degree of confidence, that the stock market is going to fall by 50 percent tomorrow. In biological models, of insects or bacteria or even of more sophisticated agents (up to my dog), it is reasonable to postulate a Markov decision process that does not involve a calculation of the long-term future costs and benefits of alternative choices. But if every person with an ounce of intellect knew that the stock market was going to fall by 50 percent tomorrow, they would all sell their stocks today -- which would cause stock prices to fall 50 percent TODAY.

My view is that many of the failings of economic models come from our pushing too far the assumption of rational expectations: We assume an almost infinite degree of rationality, for two reasons:

1. Infinite rationality is actually mathematically much simpler to handle than bounded rationality;
2. There are arbitrarily many ways to make a mistake (be stupid); of the infinitely many kinds of stupidity, none of them has been found to be compellingly pervasive; instead, behavioral economists have shown almost an unlimited number of different ways in which people' sbehavior is not rational. Infinitely many theories are not much more useful than no theories at all.

Now that you have put some (impressive) effort into this, it seems like an efficient way to make further progress would be a long conversation with me -- especially a conversation that was grounded on questions you might have after absorbing my lectures in my first year PhD course (and the corresponding lecture notes). Most of the math in the lecture notes can be ignored -- it's just algebra or calculus -- but the prose (plus my lectures) might be just what you need to bridge the gap.

I'm very busy the remainder of this week, but might have some time the week after that to have a long conversation on this.

PS. Where are you based? I think you mentioned you are on Eastern Standard Time, and maybe you said you are in New York? If in NYC, I propose that we meet in person at the NumFocus summit in NYC that I will be attending.

On Wed, Oct 23, 2019 at 4:44 PM Sebastian Benthall notifications@github.com wrote:

My understanding is that a lot of the structure code, and maybe some of the solution code, is going to soon be replaced by dolark.

https://github.com/EconForge/dolark

Pablo was saying the other day that dolark may be more general than a Markov model because it can do models with continuous time, not just discrete time.

That makes my head hurt a little and it's possible that using the discrete-time Markov subset of that functionality would suffice for a first shot at a HARK overhaul.

I'm still trying to understand the formal constraints of models in HARK. I'm trying to translate from the economics modeling used here into frameworks I'm more familiar with, such as POMDP: https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process

My best understanding so far is:

• Agents have state in discrete time.
• At each time step, they observe their state and choose an action
• these actions will change their state in the next step and accrue utility.

In the more complex Market example...

• All actor actions and states also aggregate into a Market state at the end of each time step.
• The Market state updates according to rules
• The Market state propagates to each Agent at the beginning of each time step.

If I have the generality of the problem right it will be a lot easier work on clean structure code with dolark. I think it may be within the scope of this ticket.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/96?email_source=notifications&email_token=AAKCK76ORMIN74N3P4TH4TLQQCZSXA5CNFSM4C33HN42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECCZ7DQ#issuecomment-545628046, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKCK72RNEL5L6ZQJTP7JGDQQCZSXANCNFSM4C33HN4Q .

--

• Chris Carroll

[sbenthall]

Thank you for this. I agree with you discussion of rational expectations. It's an engaging problem for sure.

I will do as you say and watch your lectures.

It's true, I'm in NYC. I will be there Nov 2-3rd, the dates of the NumFOCUS summit. It looks like NumFOCUS registration has ended; otherwise I'd be happy to attend. If you could get me in, I'd show up. In any case, I can meet with you in person that weekend.

Just a little more background on where I'm coming from... there are a lot of techniques from AI, which intrinsically needs to reckon with bounds on rationality, that build on a framework of Markov models and decision processes. If I'm not mistaken, many of the techniques used in this library are related to those techniques, though it looks like there's an (understandable) strong preference for closed form solutions over, say, reinforcement learning techniques.

https://joshgreaves.com/reinforcement-learning/understanding-rl-the-bellman-equations/

My own research, while maybe unrecognizable to an economist as economics, is largely about trying to build economic models to address the bounded rationality issues you bring up. Because I have a computer science/cog sci background, I tend to think of this in computational terms. So, I have some early work trying to bound rationality at the level of the computational power available to the decision-maker.

However, that technique is both quite difficult in terms of the mathematical analytics and also seemingly not as relevant to a lot of the policy issues around technology regulation, where the bounds on rationality seem to be more about bounds on data access. For this, I've work out and published a model of information flow based on Pearlian causation, SIFT. Still unpublished is the "economics" section of my dissertation on "Data Games", which places SIFT in a strategic context to model some legally relevant information economics issues. (background) (cutting to the models 1 2 3 4)

I'm certainly interested in structural modeling of consumption and saving behavior and look forward to watching the lectures to get a better grasp of that, and the modeling choices made.

My own research is naturally part of what attracted me to working on this project. I for sure want to prioritize the tasks at hand and learning how your community has implemented things so far. I wanted to share these materials so that you could have a better sense of the conceptual tools I'm working with to start. Clean computational modeling of boundedly rational agents and simulations of their interactions is a necessary next step in my research agenda one way or another.

[sbenthall]

See #639

[mnwhite]

Closing. This is a major component of the HARK 1 roadmap.