audio diagnostics for MCMC 📉 🔉

[goldingn]

background

MCMC is a method for estimating parameters in (Bayesian) statistical models. It results in a bunch of multivariate time series of parameter values, that are often plotted like this:

For your inferences to be correct, those chains need to have converged on the correct distribution of values. There are some summary statistics and visualisations you can use to assess this, but none of these alone is always sufficient, and they often tell you nothing about in what way the chains have not converged.

One of the reasons that's difficult is that there are usually several of these time-series (chains) and that each one describes moving around in a high-dimensional space (4 chains of a 10-parameters model means that each time step gives a 4 different points in a 10D space, and the trajectory of the points may well be correlated). So it's difficult to get a single optimal summary statistic, or to visualise them all at once.

idea

This morning I saw this tweet:

The troubleshooting section for our new fridge has melted my heart. Imagine being the copywriter who sat there for hours trying to decipher the language of fridge...

— Claire Varley (clairepvarley) November 9, 2018

and subsequent link to this web page on diagnosing failing hard drives by their sounds.

Since MCMC chains are time-series and so are sounds, could we assess convergence of all of those parameters simultaneously by converting each of the parameters to different notes, converting them into sound waves, and playing them to the user?

If they were all happily converged, I would expect the sound to be a constant hum (@hollylkirk had the idea of making the notes for the parameters harmonize, so it sounds nice), whereas a lack of convergence would have a wavering/wobbling sound. Possibly something even weirder if there's a strange dependence to the parameters, like rotational invariance (pdf).

To diagnose which parameters are troublesome, we could create a dashboard to turn up or down the volume of each parameter.

tasks

• create examples of MCMC chains that are converged / not converged in different ways
• work out how to convert time-series into sounds (requires someone with musical knowledge)
• work out how to play that back to the user, live
• create a shiny dashboard that looks like a mixing desk
• package it all up
• provide a reference library of different sounds and what they tell you about convergence

---

Note: this idea may not be at all useful, but it'll be fun to find out!

[goldingn]

There's an experimental feature in greta that means we could even make this play whilst the model is sampling, which would be pretty cool!

[njtierney]

I love this idea!

I would love to hear about creating chains that are converged/not converged, as this is something I've struggled with in my work on whether people can visually identify convergence.

I've actually had a similar thought of including this idea of audio-convergence, but I really like how well fleshed the idea out. It also reminds me of some work in Human Factors, where things like blood pressure and other biometric recordings are played as increasing or decreasing in pitch as they change during surgery so that the surgeon can be operating and not need to read machine readings.

Keen to talk more about this 💯

[njtierney]

This blog post here might be of interest: http://www.vikparuchuri.com/blog/making-instrumental-music-from-scratch/

Also these two packages might provide clues for how to read/write/play music in R:

• seewave
• tuneR

[goldingn]

Aah sweet. I thought you'd have ideas on this topic! It looks like those two packages would be super useful.

Definitely happy to chat about unconverged chains, particularly the rotational invariance thing.

Another thought I just had: rather than having one audio channel per parameter (40 channels in the above example), it might make more sense to have one per chain (4 channels in the above) and map each chain from the high-dimensional parameter space to a lower-dimensional audio space (e.g. pitch x volume x timbre) via something like PCA. The influence of specific parameters could still be turned up or down, but it would probably be easier to hear the wobbliness like this.

[certifiedwaif]

I love this idea, and have some of the required skills. We'll see what's possible.

[coolbutuseless]

The following ridiculous idea has been kicking around my head for a while:

ggaudio(mtcars) + 
   geom_audio(aes(pitch = mpg, duration = cyl), volume = 10, side = 'L') + 
   geom_audio(aes(attack = am, sustain = am, decay = disp, release = mpg), volume = 3, side = 'R') + 
   facet_channel( ~ carname)

Don't ask me how it would work - I have absolutely no idea. :)

[RPanczak]

Oh yes ❗️ But you forgot to add + theme_daft_punk() 😉

[njtierney]

@coolbutuseless that is genius! Although I have to agree, I have no idea how that would work.

[jesse-jesse]

I think this is a pretty fun idea. And potentially actually pretty useful. humans are pretty good at picking up different ces in sound.

[noamross]

I just discovered this issue and its amazing. Did anything come of this?

[goldingn]

Nope, we all did different stuff. Take it and run with it!

[noamross]

I'm going to write down my idea here because I am almost definitely not going to get around to it. Perhaps it will be picked up next year?

My thought is that an MCMC chain is a noisy signal of an underlying piece of information, we should use measures of its quality to transform a recognizable form of information - music - so that a good chain sounds like normal music and a bad chain sounds weird.

• Wrap the SoundTouch C++ library which manipulates audio pitch and speed.
• Calculate moving averages of the MCMC chain value and autocorrelation
• Select a baseline audio music sample. One could do this with raw pitches, but it's much more fun with real music.
• Warp the audio pitch to change in time along with the chain value
• Warp the audio speed to change in time along with the chain autocorrelation
• Mix the audio signal with white noise based on the number of effective samples relative to the total samples (or relative to some good number of effective samples)
• For multiple chains, do this for multiple tracks - it would be great is one had separate tracks for different, say, choral parts. Need to find some open-source audio!

[goldingn]

Oh, that's a great idea! Should be quite achievable. Or at least it's much less nebulous than my previous thoughts :)