A Raspberry Pi based Sound Sensor to classify sounds. With the Jupyter Notebooks, Python scripts and EdgeImpulse you can train any sound classfier you want. For example, why don't train a model to detect planes flying over your house? In my case, I have chosen to build a model for snoring.
This is build and tested on Rasbian Buster.
Sleep is one of the most important but least understood aspects of our life, wellness, and longevity. Until recently, science had no answer to the question of why we sleep, or what good it served, or why we suffer such devastating health consequences when we don't sleep.
While there are many excellent consumer devices on the market to track your sleep (such as the Oura Ring or Apple Watch) they have no capability to record the sound of snoring. With this tutorial you can record your snoring and visualise it using Grafana Dashboards.
With this sound classfier to detect snoring, you can better understand your sleep. For example, is snoring worse after drinking alcohol? Would snoring be less if you do breathing exercises before going to bed?
As sleep is a highly personal activity, with this 'privacy by design' sound sensor you don't have to share your data with anyone. Also, as the machine learning inference is done on the device (or 'on the edge') there is no recording of (sensitive) sounds. You just store the results of the inference on your local device.
For this tutorial I've used the following hardware:
On the raspberry make sure you've enabled SSH.
This tutorial assumes you've some experience with Python, Machine Learning etc. It's not written for the absolute beginner.
There are three options of collecting your data:
Google's Audioset link is a large collection of audiosamples. AudioSet has 527 classes of audio that can be used to train Deep Learning models. Unfortunately the quality is not always good as one sample can contain multiple sound classes. (In our case we will see that samples can contain both snoring and other classes, such as music or silence). However this is still the best way to get (free) data in quantities large enough for training deep learning models.
To download the data I use a Jupyter Notebook provided here. You can open it in Google Colab. In Step 3 of the notebook you can change the Audio Class. A full ontology of the AudioSet classes can be found here. I would recommend to download the Class white noise. You can also download the Class snoring, but do keep in mind that the 10s samples of Snoring in AudioSet also contain other sounds such as music, talking or silence.
Data augmentation is an important part of creating Deep Learning datasets. With the script provided here you can create four augmented files based on one original file. Please find the script on my Github Gist here
EdgeImpulse is a low code platform to train machine learning models used on microcontrollers (eg. Arduino) and single board computers such as the Raspberry Pi.
There are tons of tutorials on EdgeImpulse so I won't give a full tutorial here.
To train the model I've created the following three classes:
The class Snoring can be trained with the data provided in this repo (folder snoring_wavs). The class Silence can be trained with the audio wav's you've trained and augmented yourself. The class White Noise can be trained with the data you've downloaded from AudioSet and/or recorded and augmented yourself.
In EdgeImpulse go to the 'data acquisition' tab. With the built in function you can see a soundwave and listen to the audio. You can use the 'crop' function to do data cleaning. (This is a time consuming but a necessary step!) Only with high quality, non-polluted data samples you can create a model with a high accuracy.
In EdgeImpulse I've used the following settings:
With these settings I was able to achieve a model result of 92.7%.
As we will use a Raspberry Pi (Linux) there is no need to use EdgeImpulse's EON Tuner.
If you have already installed EdgeImpulse and InfluxDb on your Pi, you can skip this step
After installing InfluxDB create a database called 'SNORING'.
Also install influxdb python client:
$ pip3 install influxdb
After training the model in EdgeImpulse, you can download it to your Raspberry using the following command:
$ cd <to directory>
$ edge-impulse-linux-runner --clean --download modelfile.eim
Also, install the edge_impulse_linux package using the following commands:
$ $ sudo apt-get install libatlas-base-dev libportaudio0 libportaudio2 libportaudiocpp0 portaudio19-dev
$ pip3 install edge_impulse_linux -i https://pypi.python.org/simple
(more info can be found here
I've provided a script sound_classifier.py to store the data in a Influx database. You can run it with:
$ python3 sound_classifier.py modelfile.eim
When starting the script, there will be some errormessages about ALSA, which you can ignore (November 2021). (The Advanced Linux Sound Architecture (ALSA) provides audio and MIDI functionality to the Linux operating system.).
After running it for the first time I got the following error:
$ Original error was: libf77blas.so.3: cannot open shared object file: No such file or directory
The solution was: $ sudo apt-get install libatlas-base-dev
(see for more info here )
I've installed Grafana (link) on my laptop. With it you can easily create a connection to the InfluxDB on your pi.
Use Grafana on your laptop to connect to the Raspberry and get the data and show the results.
That's it! The most important is that you have fun while building it.
If you are interested in combining it with data from the Oura Ring ouraring.com, please see my notebook on getting data from Oura.
What could be possible next steps?