Framing and periodicity

[sbenthall]

The PerfForesightConsumerType has two poststate_vars: aNrmNow and pLvlNow https://github.com/econ-ark/HARK/blob/master/HARK/ConsumptionSaving/ConsIndShockModel.py#L1450

However pLvlNow is set in getStates(): https://github.com/econ-ark/HARK/blob/master/HARK/ConsumptionSaving/ConsIndShockModel.py#L1665

...and not in getPostStates(). Rather, aLvlNow is updated as a poststate in the code: https://github.com/econ-ark/HARK/blob/master/HARK/ConsumptionSaving/ConsIndShockModel.py#L1706

Correcting the poststate_var field to list aLvlNow instead of pLvlNow raises an AttributeError. I guess this is because the class is depending on the automatic initialization of the poststates, to create the attribute.

[mnwhite]

Yes. Attributes named in poststate_vars are the ones that are initialized at birth, and thus also when initializeSim() is called. They are the variables that describe the agents' state at the moment that time t becomes time t+1: they're the end of time t and the beginning of time t+1, and thus the variables that are mandated to exist for a new agent who comes into existence at some time.

The permanent income level pLvlNow is a state variable in the sense that its value in time t is a function of the post-state variable from t-1 and the shock variables from time t. For an IndShockConsumerType, pLvlNow is only trivially used to make a consumption decision, as the problem is normalized by permanent income. For a GenIncProcessConsumerType, pLvlNow is a direct argument into the consumption function in a non-trivial way.

But it's also a post-state variable in the sense that it exists at the moment of intertemporal transition. Non-trivial post-state variables are determined as a function of period t state variables and period t control variables, e.g. end-of-period assets are market resources less consumption. But permanent income as a post-state variable is a trivial (identity) function of its value as a state variable: permanent income at t equals permanent income at t.

On Thu, Aug 13, 2020 at 5:36 PM Sebastian Benthall notifications@github.com wrote:

The PerfForesightConsumerType has two poststate_vars: aNrmNow and pLvlNow

https://github.com/econ-ark/HARK/blob/master/HARK/ConsumptionSaving/ConsIndShockModel.py#L1450

However pLvlNow is set in getStates():

https://github.com/econ-ark/HARK/blob/master/HARK/ConsumptionSaving/ConsIndShockModel.py#L1665

...and not in getPostStates(). Rather, aLvlNow is updated as a poststate in the code:

https://github.com/econ-ark/HARK/blob/master/HARK/ConsumptionSaving/ConsIndShockModel.py#L1706

Correcting the poststate_var field to list aLvlNow instead of pLvlNow raises an AttributeError. I guess this is because the class is depending on the automatic initialization of the poststates, to create the attribute.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKRAFMQIYJMPEF7MN2WOK3SARMHXANCNFSM4P632Z4A .

[sbenthall]

Hmmm.

For context, I bumped into this while working on #761. I'm trying to functionalize out some of the behavior and that requires consistency as to how the functions used in simulations and variable designations are used.

If I understand correctly, you are saying:

• pLvlNow is a state variable because it is a function of past state and stochastic shocks and is an input to the determination of a period's control values.
• It is also an intertemporal transition, so is in that respect a 'poststate'

I find this confusing. If the variable is being used intertemporally, wouldn't its value in the previous period be pLvlPrev, or $p_{t-1}$, and the new value be pLvlNow, or p? It seems like some notational slippage was introduced into the code here.

I.e., why not:

• in getStates, assign a value to state variable pLvlNow based on pLvlPrev and the stochastic shocks
  • in getPostStates, assign to poststate variable pLvlPrev that value of pLvlNow.

(I expect that once the state variables are all properly listed out, it should be possible to automatically store the previous value for all of them. But that would be a separate issue.)

Also, what about aLvlNow? Can you explain why that is being set in getPostStates(), but not listed as a poststate_var?

[mnwhite]

I think I do that exact thing in some of the models, where there's a line of code in getStates:

pLvlPrev = self.pLvlNow pLvlNow = pLvlPrev*self.PermShkNow

The span of time in the code where what was pLvlNow is now pLvlPrev is extremely short. That can be moved to getPostStates if you want. In some sense it would make things more explicit, but it would introduce the bad feature that if you added pLvlPrev to trackvars, hoping it would track p{t-1}, it would instead track p_t.

aLvlNow isn't in poststate_vars for an IndShockConsumerType because it isn't needed at simulation initialization; note that it is in poststate_vars for GenIncProcessConsumerType. It's calculated here merely for user convenience. I guess it's also a slight code speedup for when the simulation is being run for a AggShockConsumerType as part of a Market being solved, and thus might be computed in parallel (to other instances of AggShockConsumerType).

On Fri, Aug 14, 2020 at 11:56 AM Sebastian Benthall < notifications@github.com> wrote:

Hmmm.

For context, I bumped into this while working on #761 https://github.com/econ-ark/HARK/issues/761. I'm trying to functionalize out some of the behavior and that requires consistency as to how the functions used in simulations and variable designations are used.

If I understand correctly, you are saying:

• pLvlNow is a state variable because it is a function of past state and stochastic shocks and is an input to the determination of a period's control values.
• It is also an intertemporal transition, so is in that respect a 'poststate'

I find this confusing. If the variable is being used intertemporally, wouldn't its value in the previous period be pLvlPrev, or $p_{t-1}$, and the new value be pLvlNow, or p? It seems like some notational slippage was introduced into the code here.

I.e., why not:

• in getStates, assign a value to state variable pLvlNow based on pLvlPrev and the stochastic shocks
• in getPostStates, assign to poststate variable pLvlPrev that value of pLvlNow.

(I expect that once the state variables are all properly listed out, it should be possible to automatically store the previous value for all of them. But that would be a separate issue.)

Also, what about aLvlNow? Can you explain why that is being set in getPostStates(), but not listed as a poststate_var?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-674141913, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKRAFKUIZQGPVPTAFUPYCDSAVNCBANCNFSM4P632Z4A .

[sbenthall]

Ok. As I said, the main thing that's needed for refactoring is consistency.

That's interesting about the tracking issue. I suppose my ideal implementation would look like this:

• At the beginning of a period, move all the state variables into storage in a state_prev dictionary
• Use those 't - 1' values as an input to the transition function, which would
  • evaluate each state and control variable in the order determined by their equations
• track all the state and control variables in every run by default

In the short term, I'd like to go ahead with having the previous state stored in pLvlPrev consistently across the library, since I don't think I'm going to be able to convince you to drop the 'poststate' construct without a very compelling implementation in hand.

[llorracc]

This is related to the discussion Seb and I had with @albop last week.

I believe we agreed to move to a system in which there was a clearly defined point at which each exogenous shock was realized and at which each endogenous decision was made.

The transition equations would define the state variables. But we would need to have a name for every state variable that exists at any point -- including some that might not have names in the mathematical formulation of the problem.

For example, in the portfolio problem:

Subperiod	State	Shock	Operation
0	b{t} = a{t-1} \RPort	\Risky	Transition
1	p{t}=p{t-1}\PermShk_{t}	\PermShk_{t}	Transition
2	y{t} = p{t}\TranShk_{t}	TranShk	Transition
3	m{t} = b{t} + y_{t}		Transition
4	a{t} = m{t} - c_{t}		c_{t} is chosen
5	\PortShare		\PortShare chosen optimally

There are several virtues of disarticulating things in this way:

1. There are no more "post-state" and "pre-state" variables. There are just the states at each subperiod.
2. The state variables are all defined by the transition equations -- there's no need to have any separate articulation of what is a state and what is a control
3. All the variables have a unique value within the period. So we don't need to have extra notation to decide whether the value in question is a "pre" or a "post" value
4. In the case of models where we are really thinking of shocks as being realized simultaneously (like the risky return, the transitory shock, and the permanent shock, the order in which we lay out the draws of those shocks in sequence does not matter for the mathematical solution, but might matter sometimes for computational purposes.

[mnwhite]

I support what Chris just wrote, with the extension that I'd like to make every single thing its own subperiod or step or whatever we call it. Then there are only three basic kinds of operations:

1) Assign: Determining a variable as a function of other variables. 2) Draw: Determining variable(s) by drawing from a distribution. 3) Choose: Determine a variable by selecting among alternatives.

In some sense, Choose is a subset of Assign: cNrm = cFunc(mNrm). When "acted out", assigning consumption based on the consumption function evaluated at market resources is just another kind of assignment. However, cFunc is not part of the agent's problem, but is instead determined as part of its solution. All Assign and Draw steps could be executed once the problem has been specified, but any Choose step only has a value when the model has been "solved" in some sense.

This is also the way to indicate to the future "magic solver" that this thing is to be optimized, and these are the variables in the information set at the time the choice is made.

Note that I specifically wrote Draw as potentially multivariate, whereas the other ones have one variable on the LHS. Any multivariable distribution can be written as a sequence of conditional multivariate distributions, but that can add work for the user if we mandate that. Somewhat related, a Choose step might have a multivariate LHS purely for computational speed reasons.

On Mon, Aug 17, 2020 at 1:40 PM Christopher Llorracc Carroll < notifications@github.com> wrote:

This is related to the discussion Seb and I had with @albop https://github.com/albop last week.

I believe we agreed to move to a system in which there was a clearly defined point at which each exogenous shock was realized and at which each endogenous decision was made.

The transition equations would define the state variables. But we would need to have a name for every state variable that exists at any point -- including some that might not have names in the mathematical formulation of the problem.

For example, in the portfolio problem: Subperiod State Shock Operation 0 b{t} = a{t-1} \RPort \Risky Transition 1 p{t}=p{t-1}\PermShk{t} \PermShk{t} Transition 2 y{t} = p{t}\TranShk{t} TranShk Transition 3 m{t} = b{t} + y{t} Transition 4 a{t} = m{t} - c{t} c{t} is chosen 5 \PortShare \PortShare chosen optimally

There are several virtues of disarticulating things in this way:

1. There are no more "post-state" and "pre-state" variables. There are just the states at each subperiod.
2. The state variables are all defined by the transition equations -- there's no need to have any separate articulation of what is a state and what is a control
3. All the variables have a unique value within the period. So we don't need to have extra notation to decide whether the value in question is a "pre" or a "post" value
4. In the case of models where we are really thinking of shocks as being realized simultaneously (like the risky return, the transitory shock, and the permanent shock, the order in which we lay out the draws of those shocks in sequence does not matter for the mathematical solution, but might matter sometimes for computational purposes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-675017367, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKRAFO43SXEECQXLHQPCTLSBFTP5ANCNFSM4P632Z4A .

[sbenthall]

Ok cool. I think this is something we are getting consensus around then.

I'm working towards making this happen.

I'm trying to start with some more basic refactoring: #760 #761 #660 These all have to do with make sure each model variable (exogenous shock, control variable, state variable) is getting accounted explicitly in each model an attached unambiguously to a bit of functionality for how it's getting assigned.

When everything has been standardized around that, it should be possible to write a "magic simulator" that uses them, along with a decision rule for the control variables.

Then "magic solvers" can be written that algorithmically find good decision rules.

[llorracc]

@mnwhite makes a good point about multivariate statistical draws, and that for computational efficiency we might want to have several LHS variables rather than one.

Let me expound a point that I forgot to make earlier, then elaborate on it in light of Matt's response.

The point was that we should refer to subperiods by the name of the product(s) returned therein, rather than by a numerical order (0, 1, etc). So the earlier figure should have been	Subperiod	State	Shock	Operation
b	b{t} = a{t-1} \RPort	\Risky	Transition
p	p{t}=p{t-1}\PermShk_{t}	\PermShk_{t}	Transition
y	y{t} = p{t}\TranShk_{t}	TranShk	Transition
m	m{t} = b{t} + y_{t}		Transition
a	a{t} = m{t} - c_{t}		c_{t} is chosen
portshare	\PortShare		\PortShare chosen optimally

If (counterfactually) the most efficient way to solve this problem were a simultaneous method that produced both c and portshare, the last line could be modified as below. And if it were necessary to draw the transitory and permanent shocks jointly (because they are mutually correlated, we could modify the representation as below

Subperiod	State	Shock	Operation
b	b{t} = a{t-1} \RPort	\Risky	Transition
PermShk_TranShk	$\psi,\theta$	\PermShkTranShkMatrix	Draw
p	p{t}=p{t-1}\PermShk_{t}	\PermShk_{t}	Transition
y	y{t} = p{t}\TranShk_{t}	TranShk	Transition
m	m{t} = b{t} + y_{t}		Transition
c_portshare	\PortShare		c and \PortShare chosen optimally
a	a{t} = m{t} - c_{t}		Transition

Then we would be able to write sentences like "In the period labeled c_portshare, both the level of consumption and the share of the portfolio invested in risky assets are jointly determined."

[mnwhite]

I'd prefer more descriptive indexing/naming, but I agree they shouldn't be absolutely numbered (at least by us; the code will look at their order and number them).

On Mon, Aug 17, 2020 at 5:11 PM Christopher Llorracc Carroll < notifications@github.com> wrote:

@mnwhite https://github.com/mnwhite makes a good point about multivariate statistical draws, and that for computational efficiency we might want to have several LHS variables rather than one.

Let me expound a point that I forgot to make earlier, then elaborate on it in light of Matt's response.

The point was that we should refer to subperiods by the name of the product(s) returned therein, rather than by a numerical order (0, 1, etc). So the earlier figure should have been Subperiod State Shock Operation b b{t} = a{t-1} \RPort \Risky Transition p p{t}=p{t-1}\PermShk{t} \PermShk{t} Transition y y{t} = p{t}\TranShk{t} TranShk Transition m m{t} = b{t} + y{t} Transition a a{t} = m{t} - c{t} c{t} is chosen portshare \PortShare \PortShare chosen optimally

If (counterfactually) the most efficient way to solve this problem were a simultaneous method that produced both c and portshare, the last line could be modified as below. And if it were necessary to draw the transitory and permanent shocks jointly (because they are mutually correlated, we could modify the representation as below Subperiod State Shock Operation b b{t} = a{t-1} \RPort \Risky Transition PermShkTranShk $\psi,\theta$ \PermShkTranShkMatrix Draw p p{t}=p{t-1}\PermShk{t} \PermShk{t} Transition y y{t} = p{t}\TranShk{t} TranShk Transition m m{t} = b{t} + y_{t} Transition cportshare \PortShare c and \PortShare chosen optimally a a{t} = m{t} - c{t} Transition

Then we would be able to write sentences like "In the period labeled c_portshare, both the level of consumption and the share of the portfolio invested in risky assets are jointly determined."

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-675117745, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKRAFJ2CQ4O4F6VUGUJJOTSBGMIHANCNFSM4P632Z4A .

[sbenthall]

The equations/rules for assignments will specify a partial order, but not necessarily a total order, on the variables.

In my view, this means we should not consider them to be numbered.

It also means that the exogenous shocks can be both "simultaneous" and at differentiated. What's important is that they can be evaluated prior to state or control variables.

[sbenthall]

I might not get to the full refactor of post-post-state glory in the next PR.

For the transitional work, @mnwhite could you tell me if aLvlNow is a state or a post-state in AggShockConsumerType ?

[albop]

I'm getting back to this discussion by pure chance. There is one argument I didn't convey properly.

In my view, the definition of a subperiod does not result from a succession of operations, but rather by a change n the state-space that does not correspond to a change in the information set (information set being indexed by t). It defines the succession of functional spaces that you are going to go through when doing your time-iteration.

When you do $m{t} = b{t} + y_{t}$ you move to a new subperiod, not because there is a new subperiod (there isn't) but because the state-space has changed from $b_t$ to $m_t$. It defines how you iterate "backward": i..e you start by functions of m_t (c(m_t), then functions of b_t (psi(b_t), p(b_t), ...). Conceptually, all these function could be computed as a function of m_t or b_t, but we would loose useful information.

So this is not the same as defining an order in which you compute the transitions (b then p then y then m). It is possible to keep this order to compute transitions from one subperiod to the next. When doing time-iteration, if there is a system of several functions to solve for, which have the same state-space, one can also look for the right succession of operation (either provide it or autodetect it).

Now, the idea of labeling implicitly subperiod (in the sense I introduced), with the name of the last computed variable does make some sense: after you have computed m then m is the new state-space where functions are defined. But what if you have more than one state ? Which variables do you keep to define the state-space in a given subperiod ? I suppose when could name subperiod, $(m,p)$ the period obtained just after m and p have both been computed. I'm not saying it doesn't work but it requires a bit more some thought compared to the 0,1,2,3... version which certainly does.

On Mon, Sep 14, 2020 at 10:20 PM Sebastian Benthall < notifications@github.com> wrote:

I might not get to the full refactor of post-post-state glory in the next PR.

For the transitional work, @mnwhite https://github.com/mnwhite could you tell me if aLvlNow is a state or a post-state in AggShockConsumerType ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-692291303, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKJBBP2VQLTXC5UMYQLSFZ3HTANCNFSM4P632Z4A .

[sbenthall]

In my view, there is not a linear series of subperiods.

Rather, over a period at time t, there is a directed acyclic graph of variables.

In simulation, a decision rule induces the choice variables into a normal random variable with conditional dependencies on its inputs. State variables are like this as well. Exogenous shocks are the roots (no parents) of the DAG structure. Simulation proceeds down the DAG. In principle, the evaluation of two nodes at the same "depth" can be parallelized at the computational level. Nothing makes it sequential, necessarily.

When doing the solution, I expect you can work in reverse in a similar way. Start with the "deepest" nodes in the DAG, solve them, then go by backwards induction to "back" of the graph.

So I think I agree with what @albop is saying, in that what matters about a "period" is which variable's states are being sampled/solved. But calling them "subperiods" imposes a linearity on this process needlessly.

[sbenthall]

I've renamed this issue to make it clear that it is now about the new proposed architecture. I will slate it for HARK 1.0 though that may be ambitious.

[albop]

Actually I find the term subperiod confusing. I prefer post-state which is less ambiguous. How to generalize it ? Post-post-state or post-k-state or just k-state ?

[sbenthall]

If I understand correctly from @mnwhite , "post state" refers to any variables (state, control, shocks) that need to have values at the start of period t in order to resolve the downstream variables.

One major function of the "post state" designation has been to mark variables that need to be initialized with some value when an agent is "born". [This gets in HARK's multiple agent simulation paradigm, which includes mortality. I believe this paradigm is another extension beyond the MDP formalism].

Because these variables are always evaluated/initialized before the period t, I propse that we call them "antecedent variables" instead of "post states"--as they are neither always "post" nor always "states".

I think that @albop you may be using "post-state" slightly differently--I mean, with something different in mind--in your most recent comment.

Maybe it would be better to decouple the terminology around the model (which contains variables, which may be state, shock, or control variables) and the terminology around the solution/simulation algorithms (which may approach the problem of solving or simulation a model in sequential stages).

"period" I tend to use unambiguously to mean that which is denoted by the discrete time parameter t which has special meaning within the MDP formalism.

[albop]

@sbenthall : I've been trying to formalize a bit the representation I had in mind, in order to clarify the terminology (sorry it's a bit long)

1. indexing of time: according to stochastic process convention (in which mdp theory is formulated), time t indexes the filtration of the process 𝒥ₜ, aka as the information set available at each time. With this convention time is defined as the flow of information. There can be states, choices as a function of these states, choices as a function of the first choices. All of them should be indexed by t if, knowing the solution, they can be deduced, from information available at date t . time and date are exact synonyms here.

   a. when the flow of information has a special structure, there can be special notations: for instance t+ denotes the information available at any date strictly greater than t. Or if there is a certain periodicity, one could find useful to use t+1/2 or t+1/3 or something like t_spring, t_summer... b. a period, is the interval between two dates t1 and t2 with t2>t1, something like [t1, t2[. allowing for empty periods t1=t2 is imho a bad idea, which will ultimately create a time-paradox or foster the the rise of populism. When it is not ambiguous, a period can be indexed by its starting date so we can refer to period t as a shortcut for [t,t+1[. But in principle the idea that a period lasts for one unit of time is purely conventional c. a subperiod, is a subinterval of a given period: [t,t+1[ is a subperiod of [t,t+4[ or [t,t+1/2[ is a subperiod of [t,t+1[ d. in discrete time, one can define natural ("atomic") periods as intervals [t1, t2[ such that F_t = F_t1 for any t in [t1,t2[ and F_t1 ⊄F_t2 . With these notations, t+1 is "one unit of time later", but does not necessarily denote the "next" period, as in "the next time information will be released.

2. notional periods a. within a given atomic period, [t1, t2[, one introduce new notional dates for instance sin ]t1,t2[ and use them use them to specify a specific succession of calculations. For instance if a[t1] and c[t1] are known, compute m[s] = a[t1] - m[t1] then a[t2] = m[s]*r b. this is not about notation: I like to denote notional periods explicitly like m[t$1] or with labels like m[t$consumption], but for additional flexibility I wouldn't have a problem with denoting by t$m a notional time which is defined by the minimal computations needed to compute m. This could be extended to something like t$(m,c) as the notional state, defined by m and c. c. in the basic example from a., there is no gain w.r.t. specifying that calculations must be done one after another (which a smart computer could determine on its own.. But metadata can be attached to s for instance a domain for all controls defined in s (like D(s) = m[s]\in[0,\infty[). Thanks to these metadata there is additional to perform, for instance a time-iteration algorithm. d. a good way to illustrate the difference between notional periods and basic indexes, is a maximization over two variables: c[t],psi[t] = argmax (some stuff). It is convenient to take the max over psi[t] then over c[t] rather than both at the same time. This can be done by defining: psi[s] = argmax( ... stuff at t+1 ) then c[t] = argmax(some stuff at s) . It is essentially a hint for the solver to solve for the complex max_{c,psi}, with some control over the intermediary function and domain which is defined at s . A clear difference with actual periods though is that you can swap c and psi without changing the model (solve for c than for psi). e. sometimes economists make an actual interpretation of what happens within a notional period. For instance, they say that capital is accumulated during [t,t+1[ and give a name to the end of period capital (the denote it k[t+1] but what it would be better to write it k[(t+1)-] because it is chosen based on the information known during [t,t+1[ f. this is not the same as playing with the model timing. The "timing" is the description of how decisions are taken and information revealed, not how the economist solves it

In the above I've been using specific vocabulary which partially conflicts with the terms we've been using so far. In my mind the post-state is essentially what I have described as a notional state. And I gather "sub-period" has been used either as a synonym of a notional-date or as a slightly more general notion that I don't really grasp. I kind of feel strongly that whichever naming scheme we choose should at least be compatible with the ones in 1., that I think are quite universal. For this reason, I would avoid talking about sub-periods.

The spirit of my proposal was that after you define the exogenous process, you have defined the actual time structures (atomic periods) and that suitable notation shortcuts (namely filtering on exogenous values) where enough to define the "model". Then notional dates, or sub-states, or ..., are used to describe a specific solution method.

In the table above with the list of operations, (transition, choice, random), I perceive a desire to unify everything in one description of what happens between t1 and t2, whatever t1 and t2 represent. I'm a bit skeptic about it for now, though I can't say for sure it won't work. It is enough for a RL algorithm after all. I suspect that operationally you would end up reconstructing a structure set similar to what I described. In this case, the definition of time is a result of the model (for instance, some transitions do t->t+1 some others don't).

[sbenthall]

You have introduced both mathematical concepts and notation that I have been unfamiliar with. Please forgive me if I am slow to understand and so push back in ways that may seem obvious to you.

You bring up "filtration". I'm afraid I'm unfamiliar with this idea. Looking it up, and looking up stochastic processes, I believe I see t as the index into the index set, which are the natural numbers in most cases we consider here.

When discussing an MDP, we typically formalize it in terms of a set S of states, A of actions, R of rewards, and P, the transition probabilities of states and actions. The decision rule pi specifies a probability distribution over actions given states. This induces the MDP into a markov process over the state space S. All of this is over the index set of the natural numbers, the index is denoted with t.

This is the view from "1,000 feet (or meters) up". As is quite typical with MDPs, the state space S is the Cartesian product of many substates; the action space A is a Cartesian product over many sub-actions. The structural relationships between the substates and subactions are determined by the joint probability distribution P. Note that in this formalism, the entirety of S_{t-1} is the "post state" with respect to state S_t.

The terminology gets muddled at this point because, I must presume, Economics training is focused at a smaller resolution than this. Let us, consistently with our earlier discussion, call these 'sub-states' and 'sub-actions' "variables"; we have been calling the latter "control variables". Because the joint probability distribution P involves many independent relations between the variables, it is possible to represent their structure graphically as a Bayesian network. When this is done, it makes it clear which variables are endogenous and which are exogenous: exogenous variables are those with no control variables as ancestors.

If you are following me to this point, then I think I can precisely state my disagreement with your formulation above. If t cannot be a filtration over the structured distribution over the sub-state variables because a filtration must be totally ordered, whereas the sub-state variables are only guaranteed to be partially ordered. So the premise of your formulation is incorrect.

[albop]

which premise ?

[sbenthall]

1. indexing of time: according to stochastic process convention (in which mdp theory is formulated), time t indexes the filtration of the process 𝒥ₜ, aka as the information set available at each time. With this convention time is defined as the flow of information. There can be states, choices as a function of these states, choices as a function of the first choices. All of them should be indexed by t if, knowing the solution, they can be deduced, from information available at date t . time and date are exact synonyms here.

I'm saying time t indexes a filtration on the states S and actions A. But that the variables (sub-states, sub-actions) are not totally ordered, and so do not support a filtration.

One could attempt to impose a filtration on the sub-state/sub-action process, but that would be cumbersome and I violate the use of t in the MDP formalism.

To put it another way: it is better to think of t as a full day (September 16th), rather than a particular point in time (September 16, 12:00 AM). Many things can happen during the day (the sub-actions, the sub-state variables). But, at the lower level of resolution, I can take two actions on September 16th with different information available at each time.

Your use of "information" here confuses me a bit, to be honest. At the high level, yes, given a decision rule (S_t, At) is determined based on the information from (S{t-1}, A_{t-1}). But this is not the case as the sub-state level. For example, there is no mathematical reason that an exogenous shock, which would contribute information, needs to be resolved "before" a control variable that does not depend on it.

[sbenthall]

So, rather than a period being understood as a span between t and t+1, a period would be, simply, t.

Then, the question is what to name and whether and how to order what is happening within t.

[albop]

I really don't see the incompatibility. When you define an MDP, using SARP, you get a discrete random process $Xt$ indexed by t over the state-space S, where the information set at date date t is the cumulative history of all the past events (the filtration is just the set of all possible information sets). This random process has the markov property, that is the distribution of $X{t+1}$ is characterized by looking at $X_t$. All of this is perfectly consistent and standard. t is time and it is totally ordered. I think whatever we say should be consistent with that.

One issue it that time defined by each markov chain update (each epoch) does not directly match physical time so one might want to replace $t$ by another index that is still totally ordered: $0_spring, 0_summer, 1_spring, ....$. That one is easy.

The other problem is that some of our model reformulations are implicitly changing the definition of markov chain. Take the updating equation $m' = (m - c) r$ and assume it represents the true model. I could reformulate it with a additional transition: $a' = m-c$ then $m' = a r$. I can see the result as a markov chain with additional state a, where I need to update the MC twice to arrive to a new value of m starting from an initial m. And yet, I would still like to go from t to t+1 to match the original formulation we need to break the equivalence between a markov chain update and a unit time increment. This is why we need to add another kind of date, and find another name for the "intermediary" state. The set of possible intermediary states is not totally ordered, I agree with that. Suppose there are two choice variables $x_1$ and $x_2$. And I start defining other variables as c$x_1 and b$x_2 which are determined respectively by x_1 and x_2. Which one occurs first ? We don't know and I don't think it's necessarily a bad feature. But that is the very reason while I dislike the term "subperiod" in this context.

Incidentely, I've started to use here the term "epoch" to denote the update, of the markov chain from the flow of time. It is not 100% clarifying though as it is not immune to the kind of reformulation I just mentioned.

[albop]

1. indexing of time: according to stochastic process convention (in which mdp theory is formulated), time t indexes the filtration of the process 𝒥ₜ, aka as the information set available at each time. With this convention time is defined as the flow of information. There can be states, choices as a function of these states, choices as a function of the first choices. All of them should be indexed by t if, knowing the solution, they can be deduced, from information available at date t . time and date are exact synonyms here.

I'm saying time t indexes a filtration on the states S and actions A. But that the variables (sub-states, sub-actions) are not totally ordered, and so do not support a filtration.

One could attempt to impose a filtration on the sub-state/sub-action process, but that would be cumbersome and I violate the use of t in the MDP formalism.

To put it another way: it is better to think of t as a full day (September 16th), rather than a particular point in time (September 16, 12:00 AM). Many things can happen during the day (the sub-actions, the sub-state variables). But, at the lower level of resolution, I can take two actions on September 16th with different information available at each time.

Your use of "information" here confuses me a bit, to be honest. At the high level, yes, given a decision rule (S_t, At) is determined based on the information from (S{t-1}, A_{t-1}). But this is not the case as the sub-state level. For example, there is no mathematical reason that an exogenous shock, which would contribute information, needs to be resolved "before" a control variable that does not depend on it.

Yes, I see there is some misunderstanding about the definition of the information set. In your case, assume the transition of your markov chains rely on some i.i.d. distribution $e_t$, then the information set omega at date $t$ is all the past realizations of this distribution: $(e_0, e_1, .... e_t)$.

I have also referred in the past to the concept of indexing variables by the state of the (exogenous) system but it is not relevant to the discussion here.

[albop]

So, rather than a period being understood as a span between t and t+1, a period would be, simply, t.

I don't think it is a good idea in general. time and time-interval are not the same concept. Plus I don't see any practical gain to simplify it this way (I'd rather use a time/date, to denote the interval that follows it rather than the opposite).

Then, the question is what to name and whether and how to order what is happening within t.

I actually agree agree with that, modulo the fact that I prefer to think about it happening in [t, t1[ where t1 is the end of the atomic interval. Doesn't make any difference though. There is one thing I'd like to amend about my post from before. I wrote that a "notional" time was s in [t, t1[ but I want to clarify that two notional times $p,q \in [t,t1[$ do not have to be ranked. The immediate conclusion is that vocable "time" might be as misleading than "subperiods" for this purpose, except maybe with a fancy modifier like quantum-time, tenet-time, or money-time (hence the dollar index).

[sbenthall]

One issue it that time defined by each markov chain update (each epoch) does not directly match physical time so one might want to replace $t$ by another index that is still totally ordered: $0_spring, 0_summer, 1_spring, ....$. That one is easy.

I see what you mean here now and agree.

we need to break the equivalence between a markov chain update and a unit time increment.

Yes, exactly. Indeed, it's not exactly a "chain" when we are talking about the sub- level.

The set of possible intermediary states is not totally ordered, I agree with that. ... But that is the very reason while I dislike the term "subperiod" in this context.

I see. I apologize; I misunderstood your position on the matter. I believe we are in what is sometimes called "violent agreement".

[sbenthall]

I don't think it is a good idea in general.

Here we disagree. I'm maintaining my position as motivated by the mathematical framework that is supporting the optimization guarantees. Some set of states S is being repeated, indexed by t, in the natural numbers.

Your approach seems to be anticipating a continuous-time variation. Is that so? Is this a practical consideration at this time?

in [t, t1[ where t1 is the end of the atomic interval.

I'm sorry, this notation is unfamiliar to me. What does the second bracket mean? You lost me here.

I want to clarify that two notional times $p,q \in [t,t1[$ do not have to be ranked.

Ok. I think I am beginning to understand what you mean by "notional time", how it is similar to what Chris has called 'sub-periods'.

Am I reading you right that you are suggesting elsewhere "epoch" for this concept? To me, an "epoch" denotes something bigger than point in time. Maybe "epoch" would be a better term for time measured in the aggregate model.

While we're inventing terminology, I'd like to float "occasion" as a name for unranked intra-temporal (t) variable information sets.

[sbenthall]

Or "event" (as an alternative to "occasion")

[albop]

On Wed, Sep 16, 2020, 6:41 PM Sebastian Benthall notifications@github.com wrote:

I don't think it is a good idea in general.

Here we disagree. I'm maintaining my position as motivated by the mathematical framework that is supporting the optimization guarantees. Some set of states S is being repeated, indexed by t, in the natural numbers.

Your approach seems to be anticipating a continuous-time variation. Is that so? Is this a practical consideration at this time?

Well, in the Approximate Dynamic Programming book you can read : "A common error is to assume that when you model a dynamic program in discrete time then all events (information events and physical events) are also occurring in discrete time (in some applications, this is the case)."

Look, if we refer to physical time decide that t+1 is t + one year and move on to define seasons, we are already using the fact that time is continuously valued at the macro scale.

And even if you didn't need it, I don't see what you would gain by calling "period" a point valued time. Just call it a time.

in [t, t1[ where t1 is the end of the atomic interval.

I'm sorry, this notation is unfamiliar to me. What does the second bracket mean? You lost me here.

I have written [a,b[ for an interval closed on the left, and open on the right. It is the same as [a,b)

I want to clarify that two notional times $p,q \in [t,t1[$ do not have to

be ranked.

Ok. I think I am beginning to understand what you mean by "notional time", how it is similar to what Chris has called 'sub-periods'.

Yes. To be clear, I don't want to call it "notional time". This was only valid in the above explanation, to distinguish it from physical time and usual t index .

Am I reading you right that you are suggesting elsewhere "epoch" for this concept? To me, an "epoch" denotes something bigger than point in time. Maybe "epoch" would be a better term for time measured in the aggregate model.

The advantage of epoch is it is not linked to physical time. We have plenty of words for that one: season, time-step, etc But I don't know what epoch should precisely mean. It could refer to the steps in the time-iteration algorithm (bad) or to each change of the markov chain internal state, or to each change in the markov chain from the original model, or each change triggered by an action... I get the feeling it is used in several ways I in different contexts.

While we're inventing terminology, I'd like to float "occasion" as a name

for unranked intra-temporal (t) variable information sets.

Why not.

—

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-693526237, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKNKOCE33EAKRVK5EGTSGDTFDANCNFSM4P632Z4A .

[albop]

To me, event is linked to the arrival of new information (like in event tree)

On Wed, Sep 16, 2020, 6:42 PM Sebastian Benthall notifications@github.com wrote:

Or "event" (as an alternative to "occasion")

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-693526495, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKK7LM4WXVY6I6NMUCTSGDTG5ANCNFSM4P632Z4A .

[albop]

Let's be mildly violent then ;)

On Wed, Sep 16, 2020, 6:27 PM Sebastian Benthall notifications@github.com wrote:

One issue it that time defined by each markov chain update (each epoch) does not directly match physical time so one might want to replace $t$ by another index that is still totally ordered: $0_spring, 0_summer, 1_spring, ....$. That one is easy.

I see what you mean here now and agree.

we need to break the equivalence between a markov chain update and a unit time increment.

Yes, exactly. Indeed, it's not exactly a "chain" when we are talking about the sub- level.

The set of possible intermediary states is not totally ordered, I agree with that. ... But that is the very reason while I dislike the term "subperiod" in this context.

I see. I apologize; I misunderstood your position on the matter. I believe we are in what is sometimes called "violent agreement".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-693518557, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKNKAC7QBZQMRI3IGPLSGDRP5ANCNFSM4P632Z4A .

[sbenthall]

I wonder if cognate terms on the subject of time have different connotations in English and French.

These are always tricky matters even within one language. Dictionary in hand...

I believe "period" in English refers to "a length or portion of time"--i.e., an interval of time--especially when the interval of time is within the occurrences of a repeated state.

"Epoch" refers ambiguously to a duration of time or the event that marks its beginning. At it's most extreme, "epoch" is more like temporal "origin" i.e. the Unix "time since the epoch". Maybe "epoch" should refer to the point "before" t = 0 when the initial states of variables are set.

In another conversation with CDC and MNW, I have convinced myself that in an infinite horizon model seeded with a steady state distribution, one has effectively erased the epoch, which might be useful in some cases.

The information set at particular (sub-) variables are, I believe, for now always discrete moments, albeit not totally ranked ones. For this reason I agree that "subperiod"--which would denote an interval--is inappropriate.

[albop]

On Wed, Sep 16, 2020, 8:39 PM Sebastian Benthall notifications@github.com wrote:

I wonder if cognate terms on the subject of time have different connotations in English and French.

In general yes. In this instance, I don't think so.

These are always tricky matters even within one language. Dictionary in hand...

I believe "period" in English refers to "a length or portion of time"--i.e., an interval of time--especially when the interval of time is within the occurrences of a repeated state.

Yes. I agree, its periodic nature differentiates it from a basic interval. So [t, t+1) is a period, which can be split if you want into subperiods [t,t1) and [t1,t). ( in this context subperiod is clearly defined) t on the other hand is a time or point time. Indexing period [t,t+1) by t is fine, provided you know the length of the period, calling point t a period sounds wrong to me. "Date" is slightly ambiguous as it refers to a duration in vernacular language (a day) but to a point time in math and modeling (if unit time is a year I have never heard it be referred as a date)

"Epoch" refers ambiguously to a duration of time or the event that marks

its beginning. At it's most extreme, "epoch" is more like temporal "origin" i.e. the Unix "time since the epoch". Maybe "epoch" should refer to the point "before" t = 0 when the initial states of variables are set.

Yes, it has the same connotations in french too : à l'epoque (at that old time) and "la belle époque" which refers to a time span.

It is used in the literature on machine learning (a pass through the whole dataset ) where it is linked to algorithmic events, which is what made me think of it. It also appear in the RL world too (time at which decisions are made). The latter is closest to our use, but does not clearly establish a distinction from physical time. Way I understand it, you would have epochs at dates t0, t1, .. tn . We could decide that we index epochs by their rank (epochs 0, 1, n with values in N) and times with values in in R. This would just be a convention.

This is at odd with the other meaning as the origin of time though.

On hindsight epoch might not a particularly good names for the shadow times we are looking for. Transient state, occasion, event could all be considered.

I get the feeling that we agree on the definitions, which is still a very good start.

n another conversation with CDC and MNW, I have convinced myself that in an

infinite horizon model seeded with a steady state distribution, one has effectively erased the epoch, which might be useful in some cases.

yes, when "epoch" means the beginning of time

The information set at particular (sub-) variables are, I believe, for now always discrete moments, albeit not totally ranked ones. For this reason I agree that "subperiod"--which would denote an interval--is inappropriate.

yup

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-693587805, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKMTM26RFPKZ4N2NNATSGEA4RANCNFSM4P632Z4A .

[albop]

@llorracc , @mnwhite , @sbenthall : a substantial amount of this discussion was spent dealing with the fact that "subperiods" didn't necessarily come in a precise order if we define them as the result of some intermediary calculations. Is there an actual model where this is the case ? Or does it break the possibility to perform backward induction ?

[sbenthall]

Noting here the @llorracc is suggesting:

• this could be a 2.0 feature
• a robust demo on the ConsIndShockModel would be a good first step

[sbenthall]

So [t, t+1) is a period, which can be split if you want into subperiods [t,t1) and [t1,t).

But t is almost always a natural number.

calling point t a period sounds wrong to me.

Hmmm.

machine learning (a pass through the whole dataset )

Good point. I had forgotten about this.

the RL world too (time at which decisions are made)

For on-line RL though, yes? Because one makes a decision after an epoch of training?

If this term is already used in the ML and RL literature, maybe we should avoid overloading it.

Eventually, we will want to use ML and RL methods to fit models and train agents. We don't want to set ourselves up for confusion then.

Transient state, occasion, event could all be considered.

I think @llorracc is now trying out the term "frame" for this.

a substantial amount of this discussion was spent dealing with the fact that "subperiods" didn't necessarily come in a precise order if we define them as the result of some intermediary calculations. Is there an actual model where this is the case ?

I will admit that my insistence on this point is based in no small part on my own particular research interests.

I don't believe I've been able to convince any of you that this is an interesting direction yet, but the culmination of my dissertation work was the case that contemporary problems in the regulation of the data economy can be modeled in terms of information flows characterized in terms of structural models.

So I am lobbying for the software to be designed in a way so that it can be used to explore these kinds of models.

Or does it break the possibility to perform backward induction ?

I think this is a question that needs to be explored. My intuition says that under some reasonable conditions, backwards induction should work fine. I believe there is literature on this in computer science.

[albop]

On Fri, Sep 18, 2020, 3:42 AM Sebastian Benthall notifications@github.com wrote:

So [t, t+1) is a period, which can be split if you want into subperiods [t,t1) and [t1,t).

But t is almost always a natural number.

The definitions still work for natural numbers. You could define 1,2,3,4 to be a period then 1,2 and 3,4 would be subperiods. And, anyway, it would be nice if naming conventions for a subcategory of problems (discrete time) matched ones for the broader class (continuous time), especially when the latter matches physical experience.

calling point t a period sounds wrong to me.

Hmmm.

machine learning (a pass through the whole dataset )

Good point. I had forgotten about this.

the RL world too (time at which decisions are made)

For on-line RL though, yes? Because one makes a decision after an epoch of training?

Yes, I suppose.

If this term is already used in the ML and RL literature, maybe we should avoid overloading it.

Since RL problems are essentially equivalent to one agent stochastic control problems we would need to use it in a way that is compatible.

Eventually, we will want to use ML and RL methods to fit models and train agents. We don't want to set ourselves up for confusion then.

Agreed

Transient state, occasion, event could all be considered.

I think @llorracc https://github.com/llorracc is now trying out the term "frame" for this.

It's a rather good term.

a substantial amount of this discussion was spent dealing with the fact that "subperiods" didn't necessarily come in a precise order if we define them as the result of some intermediary calculations. Is there an actual model where this is the case ?

I will admit that my insistence on this point is based in no small part on my own particular research interests.

I don't believe I've been able to convince any of you that this is an interesting direction yet, but the culmination of my dissertation work was the case that contemporary problems in the regulation of the data economy can be modeled in terms of information flows characterized in terms of structural models.

That is incorrect. I am not unconvinced at all. I just suspected that if we look at "frames" coming from the splitting of one max operation over several argument into several max operators we are naturally getting ordered frames. And I wanted to have an example in mind where this is not the case. I can see why it would be the case for some limited information problem though.

In the basic consumption-savings model it is not the case. In fact there another condition is met that a frame does not loose information w.r.t. its parent frame.

Do you have one easy characterization of one if the models you are working on ?

So I am lobbying for the software to be designed in a way so that it can be used to explore these kinds of models.

Fair enough.

Or does it break the possibility to perform backward induction ?

I think this is a question that needs to be explored. My intuition says that under some reasonable conditions, backwards induction should work fine.

Well, there are certainly subtle restrictions on the frames you can use if you don't want to loose information of increase the size of the state-space.

I believe there is literature on this in computer science.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-694598692, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKMCOSCKJGRD3QNGCOLSGK3GXANCNFSM4P632Z4A .

[sbenthall]

The definitions still work for natural numbers. You could define 1,2,3,4 to be a period then 1,2 and 3,4 would be subperiods. And, anyway, it would be nice if naming conventions for a subcategory of problems (discrete time) matched ones for the broader class (continuous time), especially when the latter matches physical experience.

I would certainly not deny that intervals are well defined on the natural numbers.

There is, in the natural numbers, a minimum non-degenerate interval size, 1, which cannot be subdivided. Maybe this is what you meant by an "atomic period".

Speaking strictly from the point of convention:

• The MDP formalism defines the relationship between States and Actions ordered by their atomic periods. Because of how this formalism is defined, the problem at t is isomorphic to the problem at t+1, and there is not much redundant information. It's a natural way to break the problem down.
• It may be possible to transform an MDP into a different "time-scale" by aggregating N atomic periods into larger intervals. It's not clear why anybody would do this.
• Time at the sub-atomic level simply does not work in the same way that it works at the atomic level. It is different because: (a) it is not totally ordered, and (b) it has no periodic structure--i.e., it is not repeating itself frame by frame.

I hesitate to assert the following, as it may muddy the waters, but: "Physical experience" is a bit of an oxymoron. The phenomenology of time is quite different from the social consensus around it, and indeed if we were truly physicists we concede time to be much more complex than the real number system (i.e., "space-time"). So I don't think "physical experience" is solid ground for choice of mathematical formalism in this case.

I just suspected that if we look at "frames" coming from the splitting of one max operation over several argument into several max operators we are naturally getting ordered frames.

Considering a DAG of variables, as determined by the transition function, the following will be true:

• There exists at least one valid linear ordering of the variables based on topological sorting.
• If we consider "framings" to be a partitioning of the variables and each subset in the partition a "frame", there will exist a framing such that the frames will have a unique topological order.

Do you have one easy characterization of one if the models you are working on ?

Not yet. Thank you for the prompt to derive one.

The difficulty I have had is that I have not been able to find the mathematical definition of the classes of models that "Economists find interesting", in this context. I am now digging for a formal description of the scope of heterogeneous agent models (HAM), as opposed to ABMs, and MDPs, but have not found one yet.

My hope is that as the formal constraints crystalize in the software, it will be expressive enough for me construct such a model.

[albop]

On Fri, Sep 18, 2020 at 4:00 PM Sebastian Benthall notifications@github.com wrote:

The definitions still work for natural numbers. You could define 1,2,3,4 to be a period then 1,2 and 3,4 would be subperiods. And, anyway, it would be nice if naming conventions for a subcategory of problems (discrete time) matched ones for the broader class (continuous time), especially when the latter matches physical experience.

I would certainly not deny that intervals are well defined on the natural numbers.

There is, in the natural numbers, a minimum non-degenerate interval size, 1, which cannot be subdivided. Maybe this is what you meant by an "atomic period".

Speaking strictly from the point of convention:

• The MDP formalism defines the relationship between States and Actions ordered by their atomic periods. Because of how this formalism is defined, the problem at t is isomorphic to the problem at t+1, and there is not much redundant information. It's a natural way to break the problem down.
• It may be possible to transform an MDP into a different "time-scale" by aggregating N atomic periods into larger intervals. It's not clear why anybody would do this.
• Time at the sub-atomic level simply does not work in the same way that it works at the atomic level. It is different because: (a) it is not totally ordered, and (b) it has no periodic structure--i.e., it is not repeating itself frame by frame.

I hesitate to assert the following, as it may muddy the waters, but: "Physical experience" is a bit of an oxymoron. The phenomenology of time is quite different from the social consensus around it, and indeed if we were truly physicists we concede time to be much more complex than the real number system (i.e., "space-time"). So I don't think "physical experience" is solid ground for choice of mathematical formalism in this case.

I just suspected that if we look at "frames" coming from the splitting of one max operation over several argument into several max operators we are naturally getting ordered frames.

Considering a DAG of variables, as determined by the transition function, the following will be true:

• There exists at least one valid linear ordering of the variables based on topological sorting https://en.wikipedia.org/wiki/Topological_sorting.

• If we consider "framings" to be a partitioning of the variables and each subset in the partition a "frame", there will exist a framing such

• that the frames will have a unique topological order.

Problem is not whether frames are clearly and uniquely defined by transitions but which set of frames are useful.

Do you have one easy characterization of one if the models you are working on ?

Not yet. Thank you for the prompt to derive one.

The difficulty I have had is that I have not been able to find the mathematical definition of the classes of models that "Economists find interesting", in this context. I am now digging for a formal description of the scope of heterogeneous agent models (HAM), as opposed to ABMs, and MDPs, but have not found one yet.

My hope is that as the formal constraints crystalize in the software, it will be expressive enough for me construct such a model.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-694885374, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKM7COACBA2KLMATVDTSGNRXVANCNFSM4P632Z4A .

[albop]

On Fri, Sep 18, 2020 at 4:00 PM Sebastian Benthall notifications@github.com wrote:

The definitions still work for natural numbers. You could define 1,2,3,4 to be a period then 1,2 and 3,4 would be subperiods. And, anyway, it would be nice if naming conventions for a subcategory of problems (discrete time) matched ones for the broader class (continuous time), especially when the latter matches physical experience.

I would certainly not deny that intervals are well defined on the natural numbers.

There is, in the natural numbers, a minimum non-degenerate interval size, 1, which cannot be subdivided. Maybe this is what you meant by an "atomic period".

I defined "atomic period" earlier in this discussion as a continuous interval that doesn't contain any new information. I'm not pushing for this vocable by the way, just clarifying what I meant. With this definitions, frames can live within one atomic period.

Speaking strictly from the point of convention:

• The MDP formalism defines the relationship between States and Actions ordered by their atomic periods. Because of how this formalism is defined, the problem at t is isomorphic to the problem at t+1, and there is not much redundant information. It's a natural way to break the problem down.

There is an alternative formulation of the Markov property where the future distribution of of a process can be forecast using (Xt, X{t-1}, ... X_{t-k}) with some finite k. By redefining the states, it can be recast as a one-period memory process but when formulating a model it is sometimes easier to refer to states that are not in the immediate past.

• It may be possible to transform an MDP into a different "time-scale" by aggregating N atomic periods into larger intervals. It's not clear why anybody would do this.

seasonality for instance. As a modeller you might want to refer to t+1 as the next quarter, while t+4 would be the next year.

• Time at the sub-atomic level simply does not work in the same way that it works at the atomic level. It is different because: (a) it is not totally ordered, and (b) it has no periodic structure--i.e., it is not repeating itself frame by frame.

I hesitate to assert the following, as it may muddy the waters, but: "Physical experience" is a bit of an oxymoron. The phenomenology of time is quite different from the social consensus around it, and indeed if we were truly physicists we concede time to be much more complex than the real number system (i.e., "space-time"). So I don't think "physical experience" is solid ground for choice of mathematical formalism in this case.

This is a quite useless comment. By physical experience, I mean day to day life, not general relativity or Planck units.

I just suspected that if we look at "frames" coming from the splitting of

one max operation over several argument into several max operators we are naturally getting ordered frames.

Considering a DAG of variables, as determined by the transition function, the following will be true:

• There exists at least one valid linear ordering of the variables based on topological sorting https://en.wikipedia.org/wiki/Topological_sorting.
• If we consider "framings" to be a partitioning of the variables and each subset in the partition a "frame", there will exist a framing such that the frames will have a unique topological order.

Do you have one easy characterization of one if the models you are working on ?

Not yet. Thank you for the prompt to derive one.

The difficulty I have had is that I have not been able to find the mathematical definition of the classes of models that "Economists find interesting", in this context. I am now digging for a formal description of the scope of heterogeneous agent models (HAM), as opposed to ABMs, and MDPs, but have not found one yet.

My hope is that as the formal constraints crystalize in the software, it will be expressive enough for me construct such a model.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-694885374, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKM7COACBA2KLMATVDTSGNRXVANCNFSM4P632Z4A .

[sbenthall]

1. d. in discrete time, one can define natural ("atomic") periods as intervals [t1, t2[ such that F_t = F_t1 for any t in [t1,t2[ and F_t1 ⊄F_t2 . With these notations, t+1 is "one unit of time later", but does not necessarily denote the "next" period, as in "the next time information will be released.

What is "F" in this notation?

There is an alternative formulation of the Markov property where the future distribution of of a process can be forecast using (Xt, X{t-1}, ... X_{t-k}) with some finite k.

True.

seasonality for instance. As a modeller you might want to refer to t+1 as the next quarter, while t+4 would be the next year.

I see. Yes, seasonality. This case has been quite tricky.

My impression of this discussion so far is that there are many equivalent ways to model the seasonality problem, and that the sticking point is computational tractability.

My own preference would be, in terms of setting up the model:

• reserve t (time, time step) as the natural numbers, scoped such that the k=1 Markov property holds
• a different variable, such as c, be used to represent the number of time steps within a cyclical superperiod (i.e, c = 4 for four seasons, where one time step (t) is a season.)
• a different variable, such as s, advances deterministically as t mod c.

But then have the solver and simulator know how to handle these cases intelligently and not waste time on, say, computing what happens if s(t+1) = s(t) + 2 because that never happens.

I.e., separate the model definition and computational levels of abstraction cleverly.

This is a quite useless comment.

I will wait at least a decade before trying to debate this point with you further. But at that point, interplanetary macroeconomics may be viable and I would like to revisit the topic.

Problem is not whether frames are clearly and uniquely defined by transitions but which set of frames are useful.

Ah. This gets to the heart of the matter.

Is there a way to formulate what "usefulness" is?

If so, maybe there's a way to programmatically discover the best framings of a particular model.

[sbenthall]

And I wanted to have an example in mind where this is not the case. I can see why it would be the case for some limited information problem though. In the basic consumption-savings model it is not the case. In fact there another condition is met that a frame does not loose information w.r.t. its parent frame. Do you have one easy characterization of one if the models you are working on ?

Trying to get at this more directly... suppose you have a problem with several control variables:

• consumption, informed by current market assets
• endogenous labor quantity, informed by an exogenous physical energy level
• risky asset allocation share, informed by an exogenous and intermittent selective attention to stock prices.

I wonder if it matters whether:

• (a) these variables are determined in sequential order, each made with knowledge of the preceding choices, or
• (b) if the variables are determined only in light of the state at t-1.

Maybe the consumer can only investigate their asset allocations on days that they don't labor.

[albop]

On Fri, Sep 18, 2020, 4:47 PM Sebastian Benthall notifications@github.com wrote:

1. d. in discrete time, one can define natural ("atomic") periods as intervals [t1, t2[ such that F_t = F_t1 for any t in [t1,t2[ and F_t1 ⊄F_t2 . With these notations, t+1 is "one unit of time later", but does not necessarily denote the "next" period, as in "the next time information will be released.

What is "F" in this notation?

The infamous filtration ;-) in other words a sigma-algebra used to measure information up to date t. In France we usually call it a "tribe" and there is a recurring joke that most annoying questions in a math seminar can be answered by "it is a custom of my tribe".

It is the standard mathematical apparatus to define random processes both in discrete and continuous time.

There is an alternative formulation of the Markov property where the

future distribution of of a process can be forecast using (Xt, X{t-1}, ... X_{t-k}) with some finite k.

True.

seasonality for instance. As a modeller you might want to refer to t+1 as the next quarter, while t+4 would be the next year.

I see. Yes, seasonality. This case has been quite tricky.

My impression of this discussion so far is that there are many equivalent ways to model the seasonality problem, and that the sticking point is computational tractability.

Not necessarily. There is a difference between the modeling language, and the convention we take and the internal representation. The internal representation can be a markov chain while the user-facing formalism could be expressed differently.

My own preference would be, in terms of setting up the model:

• reserve t (time, time step) as the natural numbers, scoped such that the k=1 Markov property holds
• a different variable, such as c, be used to represent the number of time steps within a cyclical superperiod (i.e, c = 4 for four seasons, where one time step (t) is a season.
• a different variable, such as s, advances deterministically as t mod c.

But then have the solver and simulator know how to handle these cases intelligently and not waste time on, say, computing what happens if s(t+1) = s(t) + 2 because that never happens.

I.e., separate the model definition and computational levels of abstraction cleverly.

Yes it is the kind of of things we need to differentiate physical time and computational ticks.

This is a quite useless comment.

I will wait at least a decade before trying to debate this point with you further.

But at that point, interplanetary macroeconomics may be viable and I would like to revisit the topic.

By then quantum computing might be a thing which will probably allow for conflicting opinions to be simultaneously true.

Problem is not whether frames are clearly and uniquely defined by

transitions but which set of frames are useful.

Ah. This gets to the heart of the matter.

Is there a way to formulate what "usefulness" is?

No that is to be defined. It is probably not about simulating a model though but rather about specifying additional informations to solve it.

If so, maybe there's a way to programmatically discover the best framings of a particular model.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-694912562, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKO3M4X7BOCS3WSU4QTSGNXI7ANCNFSM4P632Z4A .no

[albop]

Yes it is this kind of example which could make a difference. Some quick comments :

• if it's about changing the order of arrival of new information or about taking all decisions at date t, to me it is a problem of type 1, i.e. about cycles and about language shortcuts to define and redefine them
• if it's about ignoring selectively some shocks when taking some decisions it becomes about "decision frames" (I actually like the addition of "decision" there)

On Fri, Sep 18, 2020, 5:22 PM Sebastian Benthall notifications@github.com wrote:

And I wanted to have an example in mind where this is not the case. I can see why it would be the case for some limited information problem though. In the basic consumption-savings model it is not the case. In fact there another condition is met that a frame does not loose information w.r.t. its parent frame. Do you have one easy characterization of one if the models you are working on ?

Trying to get at this more directly... suppose you have a problem with several control variables:

• consumption, informed by current market assets
• endogenous labor quantity, informed by an exogenous physical energy level
• risky asset allocation share, informed by an exogenous and intermittent selective attention to stock prices.

I wonder if it matters whether:

• (a) these variables are determined in sequential order, each made with knowledge of the preceding choices, or
• (b) if the variables are determined only in light of the state at t-1.

Maybe the consumer can only investigate their asset allocations on days that they don't labor.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/798#issuecomment-694931224, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDSKKKXRPM66736CAYPCLSGN3L3ANCNFSM4P632Z4A .

[sbenthall]

It is getting to the point where it will be possible to implement a general framework for this. I expect it will be viable to work on it soon after #836 is pulled.

I'm wondering, from a software design standpoint, how we would like to support this in HARK.

My preference would be for this architecture to ultimately be in HARK.core.

My understanding is that the status quo is to support the overwriting of the core functionality for custom models in cases like this. See the work of @Mv77 on the new multi-stage portfolio model, this comment from @mnwhite, and the work in PR #832

Recall that the goal for this work is twofold:

• to support the frame-based organization of a model
• to allow the user to write a model configuration that is not "hard coded" as a model class.

I'm somewhat ambivalent between trying to modify the core first and then refactoring all the existing models to fit, vs. the perhaps more conservative strategy of making minimal changes to the core while subclassing AgentType to create a class with the desired API. This latter approach was what I was aiming for when working on #696.

Speaking of #696, I wonder how its experimental approach to dictionary-based model configuration looks in light of the more recent discussions in this thread. While in that experimental PR each state has its own individual transition function--because the dictionary of transition functions uses single variable names as keys--it would also be possible for it to use tuples of variables names as keys, allowing for transition functions that determine multiple variables 'at once'.

[sbenthall]

I'm now trying to figure out a generic data structure for the frame architecture (this is informed by work on #862). @Mv77 wondering if you could weigh in on this especially.

I think we have consensus about what a frame is. The question is how to design the implementation so that it elegantly does all the things we want it to do.

This is a list of desiderata for what a Frame implementation might have: (1) A Frame should know what model variables are updated when it is simulated. Name suggestion: target (2) A Frame should know what model variables it depends on when it is simulated. (3) A Frame should know the transition function. This is used in simulations. (4) A Frame should ideally know how to differentiate its own transition function. This is used in solutions. (I could be wrong here ... could somebody please confirm? @llorracc @mnwhite )

I have been playing with implementations in terms of the basic data structures that come with Python. But now I'm wondering if we should have a dedicated Frame class that can capture all this information.

The other question that's coming up for me is that when listing the variables that a transition function depends on, it needs to distinguish between current time values and previous time values for any referenced model variables.

This can be done structurally (i.e., separate lists of variable names for 'now' and previous values) or notationally (i.e., adding 'prev' to indicate a previous value). I can see pros or cons to either approach.

I have a prototype implementation ready for your review at PR #865 but it sidesteps these issues, which I do think will matter moving forward.

[Mv77]

On the data-structure question, my own uninformed prior is that the most future-proof way to do things would be for frames to have their own class. They seem like a complex enough object that at some point in the future you might want to add properties or attributes to it that might not be accommodated by primitive structures without substantial changes. It might also make it more user friendly in the sense of, e.g., knowing that the 'target' is at frame.target instead of having to remember/look up that it is on the third entry of some tuple named frame.

[sbenthall]

Thanks @Mv77 that does sound right.

The implementations in #865 and #866 don't yet use a dedicated class, but I will move them in that direction.

I think we can get the best of both worlds with a sensible constructor. I.e. initialize the Frame with something like

Frame(target, scope, transition)

[sbenthall]

Just a broader example of what the Frame initialization fragment may look like

Frame(
    target=('pLvl', 'PlvlAgg', 'bNrm', 'mNrm'),
    scope=('aNrm','Risky',"Adjust'),
    transition=transitionEquation,
    default=(0,1),
    # anything else that's needed for the solver? such as...
    arbitrage=arbitrageEquation
)

[llorracc]

One point to make here is that "arbitrage equation" is a legacy from the Representative Agent world in which one of the chief cheats is that all optimization solutions are interior solutions for which a first order condition holds. That often will not be true. (Like, in our dcegm tool). So I think I'd advocate replacing "arbitrage" with "optimality" in the above, which allows for the possibility of dcegm-like tools.

[sbenthall]

Roger that. What does 'optimality' refer to in general? A function to be optimized?