About this repo

This repo has a live app running

This repository has been created as part of my master's thesis. The thesis focuses on detecting system measures, stave measures and staves in music pages with the help of deep learning. I used two open-source datasets for this thesis. The handwritten MUSCIMA++ dataset, which has all three categories and the typeset bounding box annotations of musical measures dataset, which only contains system measures. I first enhanced the typeset dataset by using a deep learning framework called Detectron2. This has been achieved by training a model on only the stave measures from the MUSCIMA++ dataset. The trained model generalized well on the typeset data, so that I could predict stave measures from it. I took the stave measures from the predictions, created staves from them and intersected them with the existing ground truth system measures to generate more precise stave measures. After manually correcting the generated stave measure annotations, I generated stave lines from the system measures and stave measures. These have also been manually corrected where needed. With both datasets now containing all three categories, I trained several models, one for each category, one for system measures and staves, and one for all three categories combined. I also used three different backbones to compare results between them. Our results show that the models generalize well for typeset music pages, but several flaws for handwritten pages have been observed. A web-based tool has been developed and deployed to use our models which is publicly available.

With this work, I achieved remarkable results for object detection in music notes. They are state-of-the-art results and the models produced can be used by anyone by visiting the live Streamlit application or cloning, installing all dependencies and running the application locally on their own pc. To run locally, follow the installation instructions below. Running locally will download all our pre-trained models which are 8GB large.

All scripts used for this work are in the Python folder. The training, inference and evaluation scripts are probably the most important ones. The scripts demonstrate how I trained all the networks, used inference to manually check their accuracy on data and how I evaluated them based on the COCO-metric. The training process how we enhanced the typeset dataset is also available as a Jupyter Notebook in the AudioLabsEnhancement file. Other scripts are used as classes to bundle functionality together. The Streamlit application runs from the streamlit_app file.

There are two different requirement files because the requirements.txt file is used by the Streamlit application.
It is different from the local_requirements.txt file in that it does not use the integrated submodule because Streamlit does not yet support submodules.
This will be changed ASAP when the support for submodules is implemented by the Streamlit framework.

Results

Faster R-CNN with ResNet-50 backbone

Faster R-CNN with ResNet-101 backbone

Faster R-CNN with ResNeXt-101-32x8d backbone

Cloning this repository

This repository uses Detectron2 as submodule.
In order to clone the submodule correctly, you will need to use:

git clone --recurse-submodules https://github.com/MarcKletz/OMR-MeasureRecognition

If you already cloned the project and forgot --recurse-submodules,
you can combine the git submodule init and git submodule update steps by running

git submodule update --init

Installation Setup

Requirements before starting:
Python >= 3.6
to run training and testing you need a CUDA capable device and the CUDA Toolkit 10.1
you can run the streamlit app which does inference without CUDA

For Linux:

Step 1:
You will require some build / development tools, install them by running:

sudo yum groupinstall "Development Tools"
or
sudo apt-get install build-essential

Step 2:
Install python development version.

sudo yum install python36-devel
or
sudo apt-get install python3-dev

Step 3 (OS DEPENDENT):
CentOS, Amazon Linux AMI, Red Hat Enterprise Linux:
Needs cython before running the requirements install:
pip3 install cython
This is needed for pycocotools because pip apparently builds all packages first, before attempting to install them.
(ﾉ☉ヮ⚆)ﾉ ┻━┻

Ubuntu:
There are no wheels available for opencv-python-headless on some ubuntu distributions.
Instead of building it on your own, I recommend to install it with the following command.
sudo apt install python3-opencv
Dont forget to remove the opencv-python-headless requirement from the local_requirements.txt if you did this!

Debian:
Skip to Step 4

Step 4:
install all the required python libraries from this repository:
sudo pip3 install -r local_requirements.txt [-v]
This might take a while! So be patient, you may add the -v tag to see installation progress.

Step 5:
Install the Detectron2 submodule as python library by running
sudo python setup.py install
from within the Detectron2 folder.

For Windows:

Requirements:
Windows SDK
C++14.0 build tools
Microsoft Visual C++ Redistributable
can all be installed with the Visual Studio installer.

Step 1:
install all the required python libraries from the OMR-MeasureRecognition repo.
pip install -r local_requirements.txt
(Requires admin privileges!)

Step 2:
Install the Detectron2 submodule as python library by running
python setup.py install
from within the Detectron2 folder.
(Requires admin privileges, so run cmd as admin!)

Possible step 3:
If step 3 fails with an error message about an nms_rotated_cuda.cu file, try this.
add the following line in detectron2\detectron2\layers\csrc\nms_rotated\nms_rotated_cuda.cu before #ifdef WITH_HIP:

define WITH_HIP

Repead step 2.

Run the Streamlit app:

Make sure that the python package installation location is added to path, so that you can run streamlit. If the streamlit command fails with "command not found" you will need to add the following to your path:
export PATH="$HOME/.local/bin:$PATH"

Complete the installation instructions and then run:
streamlit run Python/streamlit_app.py
from the OMR-MeasureRecognition repository