This is a webpage of the course "CS423 Probabilistic Programming", which is offered at the KAIST CS department in the spring of 2019. The webpage will contain links to lecture slides and other course-related materials.
Probabilistic programming refers to the idea of developing a programming language for writing and reasoning about probabilistic models from machine learning and statistics. Such a language comes with the implementation of several generic inference algorithms that answer various queries about the models written in the language, such as posterior inference and marginalisation. By providing these algorithms, a probabilistic programming language enables data scientists to focus on designing good models based on their domain knowledge, instead of building effective inference engines for their models, a task that typically requires expertise in machine learning, statistics and systems. Even experts in machine learning and statistics may get benefited from such a probabilistic programming system because using the system they can easily explore highly advanced models.
This course has two goals. The first is to help students to be a good user of an expressive probabilistic programming language. Throughout the course, we will use a particular language, called Anglican, but we will emphasise general principles that apply to a wide range of existing probabilistic programming systems. The second goal is to expose the students to recent exciting results in probabilistic programming, which come from machine learning, statistics, programming languages, and probability theory.
It is likely that we won't have time to cover the theory of Quasi-Borel spaces in the remaining lectures. So, skip the second question in Homework 4, which is about product and function-space constructions for Quasi-Borel spaces.
We will have project presentation on the 3rd and the 5th of June. Each group will give a 15-minute talk and have a 5-min Q&A session. Here is the order of presentation.
All groups including those who do track B should submit reports on what they did and what they learnt. One report is enough for each group. The reports should have at most two pages. As usual, you can email your report to TAs and me, or submit it in the homework submission box.
We will not have a class on the 22nd of May (Wednesday). This is because I am involved in running the interviews for the graduate admission for KAIST CS.
The deadline is 6:00pm on 31 May 2019 (Friday). Submit your solutions to the TAs by putting them in the homework submission box on the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae (kwonsoo.chae@gmail.com) and Mr Hyunsu Kim (khszone02@kaist.ac.kr).
The 6th of May is a national holiday. We won't have a lecture on that day.
The deadline is 6:00pm on 10 May 2019 (Friday). Submit your solutions to the TAs by putting them in the homework submission box on the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae (kwonsoo.chae@gmail.com) and Mr Hyunsu Kim (khszone02@kaist.ac.kr).
The deadline is 6:00pm on 12 April 2019 (Friday). Submit your solutions to the TAs by putting them in the homework submission box on the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae (kwonsoo.chae@gmail.com) and Mr Hyunsu Kim (khszone02@kaist.ac.kr).
On 3 April, each project group will have to give a short presentation on its project. The presentation will have to be mainly about topic, motivation, plan and preliminary findings. Groups on track A will have to prepare 10-minute presentations, which should include time for the Q&A session. Groups on track B will have to prepare 5-minute presentations, which also should include time for the Q&A session. The presentations of the groups on track B may be slightly different and focus on giving previews on the topics that they will teach. The order of presentation is as follows:
Here is our scheme for handling late homework submissions. It assumes that the total marks are 100.
The deadline is 6:00pm on 22 March 2019 (Friday). Submit your solutions to the TAs by putting them in the homework submission box on the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae (kwonsoo.chae@gmail.com) and Mr Hyunsu Kim (khszone02@kaist.ac.kr).
Kwonsoo and Hyunsu will have TA office hours from 3:00pm to 5:00pm on every Tuesday at the room 3415 in the E3-1 building starting from 5 March.
You don't have to submit your answer. But we strongly recommend you to try it. This homework will teach you how to run Anglican.
The group project is an important part of this course. Find your project partners by 13 March 2019, and inform all of Hongseok, Hyunsu and Kwonsoo by email. Each group should consist of 3-4 students. Finally, if your group wants to go for Track B, contact Hongseok as early as possible by email.
Submit your solutions by putting them in the homework submission box in the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae (kwonsoo.chae@gmail.com) and Mr Hyunsu Kim (khszone02@kaist.ac.kr).
Studying the lecture slides and notes and the homework exercises of the course is likely to be the most time-efficient way to catch up with this course. Also, at each lecture, we will give students pointers to the related papers. If a student does not understand a certain concept, we encourage him or her to look it up in the Internet. We typically do this when we encounter a similar problem. In our case, Wikipedia, lecture notes or survey articles have helped us the most.
The next best option is to read the following draft book on probabilistic programming:
If a student feels that she or he lacks background knowledge on machine learning, we recommend him or her to have a look at the following online materials.
The online book "Probabilistic Programming and Bayesian Methods for Hackers" describes Bayesian Machine Learning using a probabilistic programming system called PyMC. Hongseok found this book easy to follow and good at explaining basics and intuitions.
A more standard reference on machine learning is Bishop's book "Pattern Recognition and Machine Learning".
Two good ways to understand probabilistic programming are to try a wide range of examples and to understand common implementation techniques for probabilistic programming languages. The following documents provide such examples or explain those techniques.
Anglican website. In particular, students will learn a lot by trying examples in the site.
Forestdb.org is a great source of interesting probabilistic programs.
Edward tutorial website and Pyro example website. Edward and Pyro are so called deep probabilistic programming languages that attempt to combine deep learning and probabilistic programming. These web pages contain interesting examples that one can try using these languages.
Goodman and Stuhlmuller's book "The Design and Implementation of Probabilistic Programming Languages". This web-based book describes the implementation of WebPPL, a probabilistic programming language on top of JavaScript. Many techniques in the book are general and apply to other probabilistic programming languages.
Frank Wood taught a graduate-level course on probabilistic programming at UBC. It emphasises on actually building efficient runtimes for probabilistic programming languages. Here is a link to his course.
A group project is a crucial part of this course. 3-4 students will form a project group, and they will carry out a project in Track A or in Track B:
Track A: A group develops an interesting application of Anglican or other probabilistic programming languages. The members of the group may attempt to find an efficient encoding of a highly complex probabilistic model (such as sequence memoizer) in Anglican, or they may develop a new probabilistic model for a complex data set and analyse the data set, or they may try to find a novel use of probabilistic programming for solving well-known existing problems (such as figuring out secret key in some security protocol).
Track B: At most two groups will be on this track. The goal of a group in this case is to study an advanced research topic on probabilistic programming or machine learning, to gain deep understanding about it, and to help fellow students acquire the same level of understanding. Specifically, a group performs an in-depth study on one of two advanced hot topics, causality and Pyro. Then, the group in this track has to teach what it learnt to other students in the course. By teaching, we mean (i) a presentation on the studied topic and (ii) a preparation of reading material and exercise problems. Also, the group has to submit a report on what they learnt. We recommend the report to be 2-to-4 pages long, although it can be longer if needed. Further information about causality and Pyro is given at the end of this webpage.
By causality, we mean attempts to discover the cause-and-effect relationships among various factors (or variables) from data (and a given set of assumptions), reason about the consequence of an intervention on such relationships (such as medical treatment), and infer useful information about counterfactuals, i.e., situations that did not happened in reality but could have happened. It is one of the emerging topics in machine learning, with a lot of blessings from leading figures from the field. In particular, bringing deep learning and causality together is one of the active research topics nowadays.
So far causality has not been studied seriously by researchers in probabilistic programming, although they two may be closely related. For instance, results on causality are often explained in terms of data-generating processes, and probabilistic programming encourages scientists to write such processes as programs. An ultimate goal of this group project is to fill in this gap slightly, find out a way of bringing causality and probabilistic programming together, and teach fellow students about what they found and learnt. Of course, this is a challenging goal, and it is ok to fail. We mentioned the goal, mainly to students in this project to navigate the big topic of causality better.
Here are a few references on causality that we recommend.
Ferenc Huszar's blog articles are very good, and introduce causality gently and intuitively. (article1, article2, article3). It is also worth reading comments of these articles. In particular, one of the comments of the first article lists recent papers on deep learning and causality.
Judea Pearl has been working on causality for many years. His interview gives you a glimpse into why he is so passionate about this topic. According to what we hear, his recent general-science book "The Book of Why" is very readable and provides good informal overview on causality. Pearl's book "Causality: Models, Reasoning and Inference" is the standard reference. Finally, it may be worth reading Pearl's recent article on causality in CACM.
Pyro is a probabilistic programming language from Uber, which attempts to support the combined use of deep learning and probabilistic programming. It is still being developed. Its syntax and implementations are not finalised yet, and change constantly. But it is one of the most exciting developments in probabilistic programming.
Pyro contains novel language primitives, such as plate, random module and tensor shape for distribution object, which can help programmers or data scientists to express or specify their intentions more clearly. Also, its implementation uses novel interesting techniques, such as poutine, automatic guide generation, and selective marginalisation of discrete random variables using the tensor broadcasting mechanism. These techniques enable efficient probabilistic inference of Pyro programs. Finally, there are cool Pyro examples, such as Attend-Infer-Repeat, that use neural networks and probabilistic programming in a non-trivial way.
The goal of this project is to study these apsects of Pyro (i.e., language design, implementation techniques, and cool examples and libraries), and explain them to the other students.
The main reference for the project is Pyro's webpage. In particular, we suggest students in the project to study the section on examples very closely. It contain information about examples, languages, implementation techniques, and other interesting libraries, such as automatic guide generation. The section even contains information about how to implement a baby version of Pyro.
Pyro and Edward from Google share several language design decisions and implementation techniques. Looking at the following papers on Edward may help students to understand these shared aspects, and also general principles for designing and implementing such deep probabilistic programming languages. However, we advise students to prioritise the study of Pyro's webpage over reading these papers.