Tutorials for Economists

[cpfiffer]

One thing I have always wanted to do is write more tutorials that empirical and theoretical economists can use to inform how they do their research. @trappmartin recently reminded me how much I have wanted to do this.

This issue is meant to be a super-issue that brainstorms possible tutorials that people might like to see on how to use Turing in the social sciences (economics in particular, but I am open to other fields if someone with experience can help).

I naturally tilt towards finance and thus most of my suggestions are likely to be finance-related. I'm hoping someone with more general econ experience can jump in here. In particular, I would like to see some macroeconomics tutorials, because that has historically been where the most sophisticated applications of Bayesian tools has been.

I also want to hear from some of the QuantEcon people to see what they think. @jlperla do you know of some people who might have some good ideas for shortish tutorials they would like to see on Bayesian methods in economics? I'm not trying to get anyone to write any of these, just to solicit some ideas.

Anyways, here are some of my ideas:

1. Trend/cycle decomposition. This is a fairly simple time series analysis that let's you talk about latent variables, analytical tricks to simplify models, and ties into a moderately large literature.
2. Structural estimation of a simple macro model, like endogenous growth or something. Probably best to use a very simple model that the literature has moved on from just for computation and expository purposes.
3. An industrial organization model, also structural estimation. I don't know this literature very well but I would like to see something on market growth and consolidation.
4. A dynamic corporate finance model -- this chapter by Strebulaev and Whited is excellent and has any number of things that could be viewed from a Bayesian perspective.
5. Models that attempt to recover latent variables, perhaps like a well-structured labor econ paper that attempts to infer skill from obervables. This might also be good because a tutorial could cover DAGs and how they can be expressed in Turing.
6. A conditional beta model from finance. Lots of evidence suggests that market betas are time varying, and it would be cool to try to estimate the posteriors for conditional betas.
7. A factor model comparison paper. Barillas and Shaken (2018) do something like this where they compare a bunch of different factor models, and I think this could be done with Bayesian model combination (where expected returns are weighted by a Dirichlet distribution) to see if the results hold.
8. Structural break tests in time series (via @rlouf).
9. Conjoint MNL/mixed logit models for marketing (via this tweet).
10. Item response theory, via this tweet. There's a good Stata blog post and an ArXiV paper on this.

Any other ideas from economist types are welcome, please add them down here.

[rlouf]

If not included in (1), a tutorial on identifying structural breaks with Bayesian methods would be great. SADF is cute but I am sure it can be greatly improved.

[jlperla]

@cpfiffer This all sounds great. Sadly, I am utterly ignorant of the practical stuff here, but maybe @donskerclass can weight in on a few applications (largely from Stan?) that you could consider porting over?

[donskerclass]

There are maybe two philosophies about going about this. Most of your suggestions are large scale fully worked examples, which could be useful for demonstrating scale and customizability to encourage users to do their own large scale projects in Turing. An alternative would be to do many much more basic models, first so that users know that there are not capabilities missing (and if there are, it's a good opportunity to, e.g., implement some new Distributions, etc) and second, because many Bayesian models are built out of combining many smaller parts, so that code examples exist to build off of and change.

On the latter front, starting points for comparison, many of which have been done already, would be the models in the Stan User's Guide and in Statistical Rethinking (1st edition has been ported to Turing, 2nd ed exists in Stan, PyMC3, and TFP so far.)

More specifically in Econometrics, a format I have seen that is useful is "take the bulk of the content of a standard 1st year grad econometrics textbook like Greene, Hayashi, or Wooldridge and redo using this new feature". Li and Racine's "Nonparametric Econometrics" is an example of this for kernel methods. Bayesian Statistics and Marketing by Rossi et al does this for Bayesian versions of standard choice models: the code is all a bunch of painstakingly written C++ Gibbs samplers so getting these models into a proper PPL would liberate them for more general uses.

In terms of the basic outline of econometrics content commonly used, I can think of the following, much of which is already implemented:

1. Microeconometrics

   • Linear models/GLMs/Panel Data. Allowing heteroskedasticity, clustering
   • IV (many types of Bayesian IV, since model is incomplete. See Rossi for examples)
   • Probit/Logit/Tobit (multiple types)/Heckit

2. Time series

   • AR/ARIMA/ETS/ARCH/GARCH (These are in Stan manual)
   • Linear State space/Stochastic Volatility (Easy to write models but sampler properties hugely important. Linear models way faster if Kalman filter can be nested. Nonlinear models like stochastic volatility are basic testing ground for particle filters vs other samplers)
   • BVARs (Keith O'Hara's C++ library and R wrapper are a good selection of models and probably speed champions right now)

3. Structural Models These are harder and you should pick a small number of ones that interest you, since they will not be small projects. Nevertheless, the following classes of models are popular, and so worth considering.

   • Demand models (multinomial logit variants from Train or Rossi all the way up to Bayesian BLP i.e. here or in Stan)
   • Dynamic decision models Rust style Nested Fixed Point approaches could be done in Turing in a way that would be hard in something like Stan since you need to fit the dynamic programming algorithm inside. Bayesian versions exist, even with latent states. You could even use the QuantEcon libraries for this. Imai Jain and Ching have the most popular Bayesian approach, but define a custom method that mixes solving and sampling so may not be easy to translate.
   • Auction models: the standard here is variants and descendants of GPV. I'm not sure what's been done on the Bayesian front: translating the nonparametrics would make a formidable Bayesian inverse problem but even a parametric valuation distribution would be a cool example.
   • Labor search models: Jean-Marc Robin style estimation procedures are nonparametric inverse problems similar in style to the auction models. With parametric latent type distributions you could do a Bayesian version.
   • DSGEs: The NYFed folks have done a lot of Bayesian DSGE work in Julia. Jesse and collaborators (including me) are working on getting models of this type into Turing, among other enhancements.

[cpfiffer]

That is quite a post, and far more than I was expecting! Thanks for the detail put in here.

I personally tend to lean more towards full tutorials than just models. We do have a model zoo for shorter models like stochastic volatility in case people just want something to copy paste, but I think for most of these topics it's worth going through data preparation and prior/posterior analysis in depth. At least, that's my preference. Plus, it might kind of take place of a modern approach to Bayesian economics/econometrics, since most of the econ-specific texts are often outdated as far as methodology is concerned.

On the latter front, starting points for comparison, many of which have been done already, would be the models in the Stan User's Guide and in Statistical Rethinking (1st edition has been ported to Turing, 2nd ed exists in Stan, PyMC3, and TFP so far.)

I think some people are already working on the second edition of Statistical Rethinking for Turing, but I'm not sure what the progress is at the moment.

More specifically in Econometrics, a format I have seen that is useful is "take the bulk of the content of a standard 1st year grad econometrics textbook like Greene, Hayashi, or Wooldridge and redo using this new feature". Li and Racine's "Nonparametric Econometrics" is an example of this for kernel methods. Bayesian Statistics and Marketing by Rossi et al does this for Bayesian versions of standard choice models: the code is all a bunch of painstakingly written C++ Gibbs samplers so getting these models into a proper PPL would liberate them for more general uses.

I think this is a good goal too. I suppose I would probably target Hayashi or maybe Wooldridge, since Greene is a pretty difficult read.

So far, a lot of the response to this issue on Twitter has been from quantitative marketing people who have (as one person noted) "rubbish tools", and there is certainly something to be said for freeing code from their archaic C++ forms.

1. Microeconometrics

* Linear models/GLMs/Panel Data. Allowing heteroskedasticity, clustering

* IV (many types of Bayesian IV, since model is incomplete. See Rossi for examples)

* Probit/Logit/Tobit (multiple types)/Heckit

All good ones. Many of these could also be canned and made single functions to be added to BayesModels.jl for later use. I also had not thought about standard error adjustments, but that's fairly easy to add to linear models. Good suggestion.

1. Time series

* AR/ARIMA/ETS/ARCH/GARCH (These are in Stan manual)

* Linear State space/Stochastic Volatility (Easy to write models but sampler properties hugely important. Linear models way faster if Kalman filter can be nested. Nonlinear models like stochastic volatility are basic testing ground for particle filters vs other samplers)

* BVARs (Keith O'Hara's [C++ library and R wrapper](https://www.kthohr.com/bmr.html) are a good selection of models and probably speed champions right now)

Also very good ones. I have long wanted to prepare a bunch of time series distributions that handle common time series cases. A discussion of state space modelling would also be very nice to see, particularly for things you can't throw the Kalman Filter on, because this is where Turing is probably more advantaged -- it would be tough to beat the exact likelihood you get from a hand-coded or canned Kalman filter.

I've also wanted to write up a BVAR tool, which would be excellent to have ready to go. BVAR is also nice because it's so structured, and we could theoretically tweak the model to get lots of performance out of it.

1. Structural Models
   These are harder and you should pick a small number of ones that interest you, since they will not be small projects. Nevertheless, the following classes of models are popular, and so worth considering.

I think whatever we can do to use the QuantEcon tools and rely on some of the spectacular tutorials/documentation over there is going to be the best. @donskerclass it sounds like a dynamic decision model might be the best fit here, would you agree with that?

I'm also quite interested in auction models, because hey, markets are rad.

As to your last point:

* DSGEs: The NYFed folks have done a lot of [Bayesian DSGE work in Julia](https://frbny-dsge.github.io/DSGE.jl/latest/). Jesse and collaborators (including me) are working on getting models of this type into Turing, among other enhancements.

I think myself and the rest of the Turing team would be interested in getting full DSGE models into Turing, and we are willing to collaborate. You can contact me if you need technical aid or are running into performance bottlenecks -- I can put you in touch with whoever you need.

I'm not sure if you're using particle sampling for your models like the DSGE folks are and I'm not sure how our particle samplers compare to their, but we have been considering updating the particle samplers recently and it might be a good impetus for us to do so.

Anyways, thank you for the suggestions! I'll add them to the list.

[jlperla]

@cpfiffer One other person who might have a suggestion, or even stan code for you guys to port is @shoshievass

[shoshievass]

I can definitely lend some ideas/code :)

1. Re: mixed logit, Jim Savage and I did a bunch of work on this and Jim has a really nice implementation w/ a bunch of best practice tricks: https://github.com/khakieconomics/rrcl

It would be awesome to have a version of this that includes an implementation of BLP. PyBLP is an awesome benchmark here -- https://chrisconlon.github.io/site/pyblp.pdf

One thing to note: a major limitation of Stan is that it doesn't currently have good ways to do inside-loop operations (e.g. contraction mappings). This is something that is actively being worked on but it is definitely a limitation.

2. Related to the previous point re: contraction mappings. A very useful thing would be to have some examples of dynamic models that combine the flexibility of Stan to draw from complex hierarchical distributions of parameters and fixed point equilibrium objects like state-dependent strategies. These are usually solved with expected maximization techniques, but I have long suspected that there is a lot of room for improvement (e.g. with a variational Bayes approach).

I'm not an expert in dynamic games but would be happy to brainstorm a good example + help

3. I have given a lot of thought re: auctions and have a bit of Stan code to donate. There is supposed to be work on this from the Stan side too in the foreseeablee future. If you're interested in this angle, please let me know. I have many thoughts and feelings (and examples) 0-:)

[cpfiffer]

Man,we're getting all kinds of great stuff. Thanks guys!

Re: mixed logit, Jim Savage and I did a bunch of work on this and Jim has a really nice implementation w/ a bunch of best practice tricks: https://github.com/khakieconomics/rrcl

It would be awesome to have a version of this that includes an implementation of BLP. PyBLP is an awesome benchmark here -- https://chrisconlon.github.io/site/pyblp.pdf

One thing to note: a major limitation of Stan is that it doesn't currently have good ways to do inside-loop operations (e.g. contraction mappings). This is something that is actively being worked on but it is definitely a limitation.

I hadn't seen those either! BLP looks very slick, and it's nice to have a robust Python implementation to look at.

Fortunately, contraction mappings shouldn't be an issue for us (I don't think), but it'd be nice to actually put them into use.

Related to the previous point re: contraction mappings. A very useful thing would be to have some examples of dynamic models that combine the flexibility of Stan to draw from complex hierarchical distributions of parameters and fixed point equilibrium objects like state-dependent strategies. These are usually solved with expected maximization techniques, but I have long suspected that there is a lot of room for improvement (e.g. with a variational Bayes approach).

I'm not an expert in dynamic games but would be happy to brainstorm a good example + help

Sure! We'd love to work on this too. I am also not an expert in dynamic games, nor an expert in static games. I'm pretty good at Risk though.

More seriously, I think this is definitely one of our more killer applications, in that if you can specify a functional form you can probably run inference on it. Dynamic models are very much something I'm interested in, and I'd like to see exactly how this might work. What would you image the simplest use case is here?

I have given a lot of thought re: auctions and have a bit of Stan code to donate. There is supposed to be work on this from the Stan side too in the foreseeablee future. If you're interested in this angle, please let me know. I have many thoughts and feelings (and examples) 0-:)

I would love your thoughts, feelings, and examples! Auction markets are very cool and I didn't even really know until I started this issue that people were using Stan to analyze them.

[jlperla]

Fortunately, contraction mappings shouldn't be an issue for us (I don't think),

That sure seems like the killer application to me... But since I am largely a "lowbrow macro applied theorist" take everything I say with a grain of salt.

There may be reasons (both "cultural" and grounded in sensible econometrics) why IO people are hesitant to use bayesian methods.

@shoshievass wasn't there an Athey et al paper where they used variational inference? If so, that might be another useful replication?

@schrimpf might also have ideas on good stuff that could fit a PPL framework?

@cpfiffer : I realized this might be more than you bargained for, and people are contributing suggestions moreso than development time. Maybe after you come up with some thoughts on feasible examples you can "put out a call" to economics and finance PhD students to help you replicate some of them? It could be a good way for students to learn bayesian methods through replication.

[shoshievass]

Man,we're getting all kinds of great stuff. Thanks guys!

Re: mixed logit, Jim Savage and I did a bunch of work on this and Jim has a really nice implementation w/ a bunch of best practice tricks: https://github.com/khakieconomics/rrcl It would be awesome to have a version of this that includes an implementation of BLP. PyBLP is an awesome benchmark here -- https://chrisconlon.github.io/site/pyblp.pdf One thing to note: a major limitation of Stan is that it doesn't currently have good ways to do inside-loop operations (e.g. contraction mappings). This is something that is actively being worked on but it is definitely a limitation.

I hadn't seen those either! BLP looks very slick, and it's nice to have a robust Python implementation to look at.

Fortunately, contraction mappings shouldn't be an issue for us (I don't think), but it'd be nice to actually put them into use.

Related to the previous point re: contraction mappings. A very useful thing would be to have some examples of dynamic models that combine the flexibility of Stan to draw from complex hierarchical distributions of parameters and fixed point equilibrium objects like state-dependent strategies. These are usually solved with expected maximization techniques, but I have long suspected that there is a lot of room for improvement (e.g. with a variational Bayes approach). I'm not an expert in dynamic games but would be happy to brainstorm a good example + help

Sure! We'd love to work on this too. I am also not an expert in dynamic games, nor an expert in static games. I'm pretty good at Risk though.

More seriously, I think this is definitely one of our more killer applications, in that if you can specify a functional form you can probably run inference on it. Dynamic models are very much something I'm interested in, and I'd like to see exactly how this might work. What would you image the simplest use case is here?

I have given a lot of thought re: auctions and have a bit of Stan code to donate. There is supposed to be work on this from the Stan side too in the foreseeablee future. If you're interested in this angle, please let me know. I have many thoughts and feelings (and examples) 0-:)

I would love your thoughts, feelings, and examples! Auction markets are very cool and I didn't even really know until I started this issue that people were using Stan to analyze them.

Awesome, let me send you a ping to discuss more. Looping in @ashdgandhi re: dynamic examples as well.

[schrimpf]

For teaching an IO course, I wrote a simple version of BLP using Turing. It doesn't include any supply side moments.

For the same course, I made a package for frequentist BLP. It's definitely not as well tested or fully featured as pyBLP.

[cpfiffer]

Fortunately, contraction mappings shouldn't be an issue for us (I don't think),

That sure seems like the killer application to me... But since I am largely a "lowbrow macro applied theorist" take everything I say with a grain of salt.

I was not aware that there was such a thing as a "lowbrow macro applied theorist", as it seems like the kind of thing you might say at a cocktail party to seem highbrow.

There may be reasons (both "cultural" and grounded in sensible econometrics) why IO people are hesitant to use bayesian methods.

Very true -- I think one of the bigger challenges of writing some of these tutorials is to keep the limitations in mind and to explain them well.

@cpfiffer : I realized this might be more than you bargained for, and people are contributing suggestions moreso than development time. Maybe after you come up with some thoughts on feasible examples you can "put out a call" to economics and finance PhD students to help you replicate some of them? It could be a good way for students to learn bayesian methods through replication.

I would love to do that! I'm certain there are PhD students who would love to learn some of this, particularly if they have someone a little more familiar with the tools and methods to watch over them.

For teaching an IO course, I wrote a simple version of BLP using Turing. It doesn't include any supply side moments.

For the same course, I made a package for frequentist BLP. It's definitely not as well tested or fully featured as pyBLP.

@schrimpf These are pretty amazing! Would you mind if we adapted the BLP with Turing example you put up?

[schrimpf]

@schrimpf These are pretty amazing! Would you mind if we adapted the BLP with Turing example you put up?

Yeah, that'd be great.

[cpfiffer]

For those of you interested in this issue, note that I haven't forgotten about this stuff. I'm taking the comprehensive exams for my PhD on July 30th - 31st, so I'm in lockdown mode for the moment. I'll get to this in earnest when I'm unemployed (fail) or done with tests forever (succeed).