search

MarlonMueller / edge-iot

Edge Computing and the Internet of Things
2 stars 0 forks source link

readme

Header

In this repository, we have implemented BirdNet, a distributed system for bird monitoring. By deploying nodes in the wild, the system captures bird sounds and classifies the detected sounds using Deep Learning. Each node periodically reports identified bird species to a gateway. Detections from these nodes can then be visualized using a dashboard.

Table of Contents

Hardware

The following equipment is required:

For ESP and RPI, make sure to have wired the different components. Please refer to docs/wiring.md. If you have made changes to code regarding the wiring of some component, wire the physical cables accordingly.

Getting Started

Before running code contained in this repository, the following prerequisites need to be fulfilled.

Prerequisites

Note that the abbreviations ESP, RPI and DEV correspond to the ones within Hardware.

Demo

Once you have checked that you fulfill all the prerrequisites, follow these steps to have a functional demo of BirdNet to detect Water Rails, Cetti's Warblers and Common Blackbirds 🐦.

  1. Clone this repository in both DEV and RPI:
git clone https://github.com/MarlonMueller/edge-iot.git
  1. In DEV device, move to the cloned repo directory. Flash the ESP code into each of your edge nodes by executing the command below. To test whether the code was correctly flashed into the devices, see the output opbtained in the terminal. If it has failed, a message error should appear.
idf.py build flash monitor
  1. For the RPI, move to the cloned repo directory. A Makefile is specifically designed to automate the compilation of the file for Raspberry Pi.
make -f Makefile.RPi
sudo ./build/main_rpi

  1. For the backend please go to https://github.com/MarlonMueller/edge-iot/tree/backend

  2. For the frontend please go to https://github.com/MarlonMueller/edge-iot/tree/frontend/bird-watcher

Training Networks

To train your own networks, an end-to-end ML pipeline can be run.

[!CAUTION] The full pipeline can currently only be executed on a Linux-based operating system due to the internals of ESP-DL. Please adhere to the suggested versions for Python, dependencies, and other components to reduce the chance of errors.

Configuration

Setup the Python environment. Example

# Create conda environment
conda create -n "birdnet" python=3.8.18 pip
conda activate birdnet
# Install requirements
python -m pip install -r requirements.txt

We apply the same audio preprocessing to both training and inference. For this, we have implemented a C++ audio processing module, which can also be invoked from Python. For its setup run

python src/audio/setup.py build

and set the correct path to the C library in audio_processing.py. Then run

gcc -x c -o src/audio/preprocess.so -D RUN_PC -shared -fPIC -I include/ src/audio/preprocess.cpp -lm

For more details please refer to docs/audio_processing.md and the source files.

Within pipeline.py first, define a Xeno Canto query. By default, the network will be trained to detect the top-3 bird species resulting from this query. Data from ESC50 is used as a negative class.

Within birdnet.py you can define your own CNN. Note that our Jinja template currently only supports Conv2D, MaxPool2D, Flatten, FullyConnected and Softmax layers. Please refer to this link before including other layers to ensure that they are supported by ESP-DL. Note that although the app partition size is extended by default, larger networks might necessitate further size adjustments.

Deployment

On a high-level, the pipeline consists of the following steps

  1. Download audio files from Xeno Canto and ESC50 if not already present
  2. Convert audio signal to normalized mel-frequency cepstral coefficients
  3. Generate TensorFlow datasets from h5 file and annotation file
  4. Train model (and optionally visualize training history)
  5. Quantize model to int8 and int16 and evaluate performance
  6. Render model to C++ code using Jinja template

and can be executed using

python pipeline.py

If successful, six model and weight C++ files will be placed in the src/model directory: three for an int8 quantized version of the trained model and three for an int16 version.

Please note that multiple intermediate models will be generated during this process, which are not relevant for the ESP32 code. To run the model on the ESP32, integrate the generated .cpp coefficient file in src/esp32/CMakeLists.txt, for example

set(srcs
    main.cpp
    ../model/birdnet_int8_coefficient.cpp
)

and include the model it main.cpp, e.g.,

#include "birdnet_int8.hpp"

Furthermore, ensure that the correct input quantization exponent input_exponent is set in main.cpp. You can find the coefficient as a comment in the model .hpp file, for instance: input_exponent: ['-7'], or alternatively in the quantization log.

Time and heap logging can be enabled inside the Kconfig.projbuild file. Ensure that the project is fully cleaned before compiling to ensure that the changes take effect.

Additionally, if modifications are made such as changing the number of species, it is necessary to adjust the ESP32 code accordingly.

For more insights, please consult pipeline.py and the respective Python modules within /src.

Repository Structure

Please note that this provides only a high-level overview. Additionally, some data is not contained on GitHub but will be generated during processing.

./
├── .github/
│   └── workflows/
│       └── continuous_integration.yml
├── assets/
├── components/
│   └── espressif__esp-dsp
├── data/
│   ├── audio/
│   │   ├── ESC-50-master
│   │   ├── annotation.csv
│   │   └── audio_preprocessed.h5
├── docs/
│   ├── reports/
│   ├── slides/
│   └── [docs.md]*
├── include/ <- C++/C header
├── src/
│   ├── audio/ <- Microphone, audio preprocessing
│   ├── dataset/ <- Xeno Canto, ESC-50, TF datasets
│   ├── esp-dl/ <- (Hotfixed) ESP-DL
│   ├── esp-led/
│   ├── esp32/ <- ESP32-S3 main.cpp
│   ├── gps/
│   ├── heap-log/
│   ├── lora/
│   ├── model/ <- Python, C++ models
│   ├── quantization/
│   ├── rpi/ <- RPI main.cpp
│   ├── templates/ <- Jinja template
│   ├── utils/ <- Plots
│   ├── websocket/
│   └── wifi/
├── .gitignore
├── .pre-commit-config.yaml
├── CMakeLists.txt
├── Makefile.RPi
├── README.md
├── pipeline.py
├── requirements.txt
├── requirements_idf.txt
└── sdkconfig.defaults

Further documentation

Please see docs/ for further interesting documentation. Of special relevance are: