Documenting the `hakaru` command

[GenevaS]

There is no explicit documentation of the hakaru command right now. What should be included on this page?

[zaxtax]

I think explaining the two ways the hakaru command can be run is a good start now, and talking about the weights when that functionality is added.

[GenevaS]

There is a second way to run the command? I only know of hakaru program.hk

[zaxtax]

Yep. It's designed for running markov chains. It's used in

http://hakaru-dev.github.io/transforms/mh/

[GenevaS]

Ah, I see it. Haven't read that far into the manual yet.

Is it as complex as the example (mh target.hk proposal.hk | simplify - | hakaru - target.hk | head)? Or can it be run as hakaru - target.hk? What is the difference between the two?

[zaxtax]

When passed two programs like hakaru transition.hk init.hk, the first program transition.hk has type a -> measure(a) while init.hk has type measure(a). What that command does is take a draw from init.hk then passing that sample to transition.hk as input, then keep passing the resulting draw into transition.hk

[zaxtax]

- is a convention for talking about what comes from standard input. In principle we could have done

mh target.hk proposal.hk | simplify - > transition.hk
hakaru transition.hk target.hk | head

[GenevaS]

So for hakaru transition.hk init.hk only one sample is drawn from init.hk (the first one) and then all other samples are from transition.hk outputs?

[zaxtax]

Yep.

[GenevaS]

Ok, got it.

Regarding -, is this a commonly known thing (i.e. should it be mentioned somewhere)? I've personally never seen it before, but I haven't worked in the command line extensively either.

[zaxtax]

It comes up in some UNIX tools like vi and grep. It's just a convention, https://unix.stackexchange.com/questions/16357/usage-of-dash-in-place-of-a-filename

[GenevaS]

Ok, not noteworthy for Hakaru then.

So this page should include a quick introduction (e.g. " The hakaru command is used to generate samples from your Hakaru model.") and then usage instructions and small examples?

[zaxtax]

Sounds good. Also the dash thing is convention but I don't think it's widely known so mentioning it probably good since we do use it in examples.

[GenevaS]

Ok. Where do you think the best place to bring it up is? This new hakaru page maybe?

[zaxtax]

Yep that hakaru page is a good place. It gets used in nearly all the commands but not enough that it makes sense anywhere else.

[GenevaS]

And the last question for now is - where should this page go? Would it go with the transformations or the introduction?

[zaxtax]

Introduction.

[GenevaS]

Alright, the first attempt at documenting the hakaru command is in commit 7726548ce12619b0cf94f9d2cadab08ec93eb4bf

[GenevaS]

I've added dash documentation in commit af0592f54158f5dbf9e9f716b2f5fc63aff37912.

Did we still want to expand this page with a description of weights in Hakaru?

[JacquesCarette]

I think so.

[GenevaS]

Other than saying "The hakaru command prints out weights for each generated sample. Sample weights are typically all 1, but some programs will cause variances.", how should I explain weights in Hakaru?

[zaxtax]

So the hakaru command is an importance sampler. Every sampler is designed to approximate the expectation of some probability distribution. Do you think it would be helpful if you included how the samples and their weights are used to compute an approximation of this expectation?

[GenevaS]

Examples are always more useful than a general description.

[zaxtax]

Maybe an example like like a biased coin like

weight(3, return true) <|> weight(1, return false)

And then using my awk script to do the count

echo "weight(3, return true) <|> weight(1, return false)" | hakaru - | head -n 2000 | awk '{a[$2]+=$1}END{for(i in a) print i,a[i]}'
false 2028
true 5972

[GenevaS]

(53f51fd48d02a845309f6b9d85f25bfb43894ba2) I've replaced the normal distribution example with the biased coin and introduced hakaru as an importance sampler in the preamble. What else needs to be done for this page do you think?

[ccshan]

I don't see how either the normal(0,1) example or the weight(3, return true) <|> weight(1, return false) example shows "how the samples and their weights are used to compute an approximation of this expectation", because the weights in both examples are constant across samples and moreover ignored by subsequent processing. So the current text fails to address @JacquesCarette's desire to "expand this page with a description of weights in Hakaru".

[GenevaS]

I am missing some conceptual information for this. I don't think that a single line about weights in the beginning is enough to explain Hakaru weights to someone else. The example is supposed to complement this information, not stand alone. It should (hopefully) make more sense once I understand how the weights work myself.

[ccshan]

Yes, so how might I help you (or anyone else) understand weights?

May I suggest you try the following two-part exercise, which helped some other people in my experience?

1. Do https://github.iu.edu/ccshan/prob2017/blob/master/0110/README.md while reading https://github.iu.edu/ccshan/prob2017/blob/master/papers/eddy-bayesian.pdf
2. Do https://github.iu.edu/ccshan/prob2017/blob/master/0112/README.md

[GenevaS]

There are a few changes here now (latest commit 851a36e10ea36ba04116b2dd75d4edf5f4d6ea82):

• I broke the awk script example into two parts, which explicitly outlines counting samples and counting weights
• Clarified how Hakaru handles weights (starts as 1.0 and is changed by other Hakaru processes/primitives)
• Added some background knowledge and a reference for Monte Carlo methods and importance sampling Other than perhaps finding an example of a Hakaru program generating different weights across samples, I don't think that there is anything else to do for this page.

[ccshan]

Other than perhaps finding an example of a Hakaru program generating different weights across samples, I don't think that there is anything else to do for this page.

Agreed. How about we replace biasedcoin.hk by the following:

i <~ categorical([0.25,0.75])
weight([2,1][i],
return [true,false][i])

[GenevaS]

Sure. Is this modelling a particular problem or is this more of a toy problem? I want to be able to tell people what is happening in the model (which is why the current example section starts off with "The "trick coin" is a basic example that is used to introduce the probability or expectation of an outcome. Suppose we are given an unfair coin that follows the distribution:")

[ccshan]

Let me adjust the numbers (but not the structure) of the program to make it match the disintegration of examples/burglary.hk, applied to alarm = true:

burglary <~ categorical([0.0001, 0.9999])
weight([0.95, 0.01][burglary],
return [true,false][burglary])

This example is discussed in Section 2 of "Probabilistic inference by program transformation in Hakaru (system description)".

[GenevaS]

Is this an accurate description? The line burglary <~ categorical([0.0001, 0.9999] makes me unsure:

"To demonstrate weights in Hakaru, a sample problem of a bulglary alarm is adapted from Pearl's textbook on probabilistic reasoning[^2]. In this problem, the burglary alarm in your home has been triggered. There is a 95% chance that a burglary is taking place, and a 1% chance that it is a false alarm. From known data, there is a 0.1% chance that a burglary is taking place and a 99.9% chance that the alarm was triggered by another event. This can be modelled in Hakaru by the program:"

[GenevaS]

Latest updates in fd0ba43eb054e96232bfb1d0de5b6a2c6bc44cac

[ccshan]

Thanks; I commented on that commit.

[GenevaS]

Should we consider this page complete for now?

[ccshan]

Fine with me!

[GenevaS]

Ok, I've deployed all the changes. Thanks for all the help!