magenta / ddsp-vst

Realtime DDSP Neural Synthesizer and Effect
Apache License 2.0
724 stars 72 forks source link

Untuned pitch detection #16

Open hargisss opened 2 years ago

hargisss commented 2 years ago

I'm a little frustrated that when I play the instrument with my theremin, it snaps to the correct pitch rather than interpreting the pitch of the theremin directly. I can't play microtonal scales or play out of key, or capture the full nuance of my performance.

I can see why you've made it this way: I imagine there are many instruments where raw pitch detection was too unpredictable and noisy to work. But I think it might work with a theremin which has a very clear signal. I'd like there to be a checkbox or toggle that switches between tuned and raw pitch detection, perhaps with a warning stating that the results might be unpredictable, especially with acoustic instruments.

It might justify the addition of an 'advanced settings' section for this and other settings, like changing the tuning of the scale, and also changing the smoothing algorithm for the pitch. Why decide on the perfect option when the user can decide?

UPDATE I've included this spectrogram to illustrate that the VST is currently auto-tuned:

unknown2

Mrugalla commented 2 years ago

I didn't participate in the development of this plugin but just so you know.. usually when developers make pitch detection algorithms they use the FFT algorithm for that, which is always a little imprecise in terms of time as well as pitch, but there are hardly any alternatives unfortunately. that's the reason why it's not possible to measure such nuanced playing in realtime. that's why no audio-to-midi software yet captures pitchbends accurately as well. autotuning pitch detection data is basically the only way to use the information from this algorithm in a way that is usable at all. however i'd agree with you that they should add features for microtonal scales, or maybe even xenharmonic ones. that would be exciting

hargisss commented 2 years ago

Good point Mrugalla - It seems the way they do this now is by using a machine learning algorithm called CREPE https://arxiv.org/abs/1802.06182 to estimate the pitch. It isn't a simple algorithm - the full version of it is capable of what I'm asking for but is probably too large to fit in the plugin (which is 600MB even with the distilled version). For most signals, that's just about the best you can do. However, I'm using a theremin as an instrument, and the signal from a theremin is a whole lot simpler than the signal of a voice or other instrument - in theory you should be able to calculate it just by counting zero crossings in the signal. A theremin mode which uses simple zero crossing detection could be useful not just for theremin players, but also for electronic musicians who could drive the plugin with a very simple synthesizer.

Mrugalla commented 2 years ago

oh yeah! that would actually be really cool. i mean switching between different pitch detection modes, like zerocrossings, fft, machine-learning based ones etc

Turkrosoft commented 11 months ago

Right, it should also support manual fine-tuning with microtonal scale adjustments, and disabling pitch detection/correction entirely.