search

kosmasK / ViolinBowTracking

M.Sc. SMC Thesis, MTG-UPF, 2015-2016
GNU General Public License v3.0
3 stars 0 forks source link

Alternate code to detect violin bow direction? #1

Open guysoft opened 6 years ago

guysoft commented 6 years ago

Hey, I was wondering - have you done any other bow detection methods? Such as feature extraction from audio? I can see you mentioned some in your paper but can't find any implementation anywhere.

kosmasK commented 6 years ago

Hey Guy,

Thank you for your question! The code included in this repo consists only of the OpenCV bow tracking that calculates 3 basic features (bow velocity, acceleration and inclination), implemented in C++. However the code in this repo needs further improvements in order to get faster and robust tracking, either in real-time, or by feeding a prerecorded video. Additionally based on the position and orientation of the Aruco marker you can extract bow-bridge distance and skewness quite easily.

Regarding the process of extracting bowing properties directly from audio features I would say that it's still a quite challenging task. You need to have a ground truth that would specify when and what was happening during the performance in order to build a good recognition model. In my thesis I used a library called Essentia for calculating low-level audio descriptors, which were compared to the computed video features (ground truth). I found that MFCC2, MFCC4, MFCC6, MFCC11, MFCC12, barkbands spread, pitch, pitch instantaneous confidence, spectral crest and Inharmonicity can help to extract bow inclination, while MFCC1, MFCC5, MFCC7, HFC, spectral crest, flux, and strong-peak are related to the bow velocity.

However, the dataset that I 've used was quite limited, considering that only one violin player was performing during all the recordings. It would be nice to see what results you could get from different performers.

What is the context of your application? What is the problem that you want to solve? It would be my pleasure if I can help by giving some guidelines.

Cheers!

guysoft commented 6 years ago

Hey, I was just curious if I could get the bow direction to control FX. I am part of a new community that builds electric violins at https://violinmakers.org . At the moment it looks like an arduino would be a better way to do this. It just seems like low hanging fruit with the phase so obvious in the waveform. However nearly all audio analysis ignores phases because they are not audible.