tutorials

[dustinvtran]

GPy has incredible tutorial notebooks. They teach basic concepts related to GPs and use GPy as a basis for explaining these concepts. I think it's a great idea, and I've personally found them useful as I dug though the GP literature.

I think we can do something similar. Each notebook instructively explains a concept. The code is similar to the end-to-end examples currently in the repo, but with more words and explanation about the various options that are used. Here are some examples of concepts to teach:

• Black box variational inference
• Mixture modeling with stochastic variational inference
• Posterior predictive check

[dustinvtran]

I am not sure if we wrote this down already, but I am doing it in case we didn't here as it's probably the most relevant location. Following discussion with Dave and Alp, it makes sense to organize the more in-depth tutorials via:

modeling

• mixture of gaussian's
• hierarchical linear model
• latent Dirichlet allocation
• variational auto-encoder

inference

• coordinate ascent VI
• SVI with local vs global latent variable updates
• BBVI
• GMO (with linear mixed models as the running example)

criticism

• PPC's
• leave-one-out CV

Also following discussion with the Google Brain folks, Rif suggests we (as in Bayesflow + Edward) should focus very much on the following trinity of model examples: mixture of Gaussian's, a hierarchical linear model, and a variational auto-encoder.

[dustinvtran]

inference

• How do we do greedy layer-wise training in Edward, where one layer is learned with the others held fixed? As another example, how do we separate learning rates for the generator network and inference network in sigmoid belief nets? Define variable scopes. Then define separate optimizers for each variable scope. Let's have a inference tutorial demonstrating this.

[dustinvtran]

inference

• model-specific inference algorithm such as poisson factorization, explaining poisson factorization and its inference, its implementation, and a demo of its use

[dustinvtran]

more tutorial ideas

model

• probabilistic decoders with priors (describing the vae example's implementation)
• maja's log linear network model

inference

• inference networks (describing the vae example's implementation)
• variational EM

criticism

• prior predictive checks (and motivate by linking to vae for wanting to do this, e.g., seeing how well the model parameters cover the full prior space)

end-to-end

• Semi-supervised learning
• Reinforcement learning
• Causal inference
• Transfer learning
• Domain adaptation and meta-analysis
• case study based on a domain application, such as uncovering topics present in documents, analyzing the firing rates of neurons, and recognizing objects in images

somewhere

• how to do CPU/GPU integration in edward (pointing mostly to tensorflow)
• how to do distributed training in edward (pointing mostly to tensorflow)
• using edward idiomatically with tensorflow

[dustinvtran]

A few tutorials are up and online now! I think we set up an excellent foundation for both organizing the structure behind the tutorials and also building the momentum in writing them.

The scaffolding of documentation is currently

1. API (doc/source/*.rst)
2. Tutorials (website/tex/*.tex)
3. Example scripts (examples/*.py)

Are we still planning to use iPython notebooks? Here's one proposal. The code snippets in the model, inference, and criticism tutorials should link to a Python script as source, which they already do. The end-to-end tutorials are hairier because we go step-by-step through the code and possibly display pretty visualizations. Therefore all end-to-end tutorials should link to iPython notebooks as source?

[bhargavvader]

Hello, I was thinking of helping by making the end-to-end tutorials as an iPython notebook. Any tips on good ways to go about doing this?

[bhargavvader]

ping @dustinvtran

[dustinvtran]

Oops, sorry the notification must have fell through the cracks.

I'll leave it up to you how to structure the notebooks. :) One suggestion is to have each code snippet as a Python block. Between the code snippets is the same text as in the end-to-end tutorials.

[bhargavvader]

Sounds good. :) I'll get at it over the weekend!

Should I create a folder called notebooks in docs and put all the notebooks there?

[dustinvtran]

Sure that sounds about right.

[dustinvtran]

Update: The organization for examples is as follows. Edward tutorials—each with a corresponding Jupyter notebook—is the gold standard. Examples in the examples/ directory are one-off scripts; their end-goal is to eventually become an Edward tutorial.

[apollostream]

I've been experimenting with Edward and find it to be a fine piece of work. Well done & thank you!

My go-to environment for data science has been R within RStudio, and I've done extensive work using Stan via R package rstan. So, I've investigated using Edward through the R packages reticulate (access Python from R) and tensorflow (access TensorFlow from R). Using those two packages I've ported a few of the Edward example/tutorial ipython notebooks to R Markdown notebooks.

I've attached a rendering of one: edward_getting_started.pdf

Not sure which of these this effort is:

• A curiosity in the vein of "Yeah, you can do it ... but why?" or

• A useful bridge between the R & Python communities, transporting R users who already use Stan for probabilistic programming from a Statistics-centric world into a Machine Learning-centric world (e.g., deep learning neural networks; GPU-based computation; etc.).

Hopefully it's more the latter than the former.

Does it make sense for me to upload some of these R Markdown notebooks here?

[dustinvtran]

Yes. That looks very helpful!