search

x42 / fat1.lv2

Fons Adriaensen's AT1 -- Autotune LV2 plugin
http://x42-plugins.com/x42/x42-autotune
GNU General Public License v2.0
72 stars 16 forks source link

MIDI output? #22

Open SpaceAgeHero opened 3 years ago

SpaceAgeHero commented 3 years ago

Hi, I'd like to record x42-autotune's "suggestions" to a midi-track for me to easily adjust afterwards. Please excuse me if this was implemented already. I've tried in Ardour but my MIDI track didn't recognize any input from the plugin. Kind regards

Schroedingberg commented 2 years ago

I would like to draw attention to this, since this would make working with x42 autotune much easier. As I understand it, this is basically what proprietary software like Melodyne does and I dare say that lack of this functionality might be a showstopper for people thinking about switching to a FOSS alternative. If I am correct, I am interested in trying to implement this. Would be my first time actually programming in the audio plugin world, but you need to get started at some point (or never). If my analysis is incomplete or wrong, please correct me so I do not start hacking based on false assumptions :D

Schroedingberg commented 2 years ago

In fact, I came here to open a similar issue, as a feature request and found this one, which is close enough. Maybe it should be renamed to "Make Melodyne obsolete" :-D

Schroedingberg commented 2 years ago

Alright, I tried to research a bit in the code. I guess adding a MIDI output in principle should be a standard task. If all fails, I can use the passthrough MIDI filter as a template. It is not obvious for me however, where the pitch estimation happens, which is what is necessary to generate the MIDI output.

x42 commented 2 years ago

It is not obvious for me however, where the pitch estimation happens,

That'd be in Retuner::findcycle. A FFT and then peak-finder (loudest freq bin).

Anyway, there are various problems with this.

1) You want to feed the MIDI note-even before the actual sound reaches the retuner and resampler. So the detected MIDI note is too late. It would have to be moved back in time for a bit.

2) process-loop. You cannot use a MIDI track to feed the plugin while at other times record onto that MIDI track. Effectively that's a feedback loop.

Also note that autotune.lv2 is a pitch-shifter / vocoder similar to Antares' autotune, it is not comparable to Melodyne at all.

Schroedingberg commented 2 years ago

Thanks, very enlightening!

Yeah, I wasn't imagining this as a real time thing. I was thinking along the lines of recording the estimated notes to an extra MIDI track, then later use the result of that (after tweaking it) and use that as the MIDI input for the actual autotuning (removing the MIDI output again). The idea is, assuming the vocals are good enough, to save some of the tedious work of recomposing the singing melody.

Assuming I have correctly understood the problems you pointed out, this sounds more feasible to me than the "feedback-loop" approach. Do you agree? :-)

(Forget about the melodyne stuff, that was polemic and probably based on a lack of exposition to the product)

x42 commented 2 years ago

Another approach would be to first run an audio-to-midi process offline and use that as input. In Ardour you could use the bundled script (https://vimeo.com/186035015)

Schroedingberg commented 2 years ago

Wow, that looks awesome. So the basic underlying problem is actually solved already :)

x42 commented 2 years ago

In recent Ardour you could also add the script to a button (or keyboard shortcut), just remember to first draw a MIDI region and select source audio and destination MIDI region. https://discourse.ardour.org/t/how-are-these-quick-buttons-called-how-to-trigger-them-by-keyboard-shortcut/106008

Hard to say how well it works for vocals. Ardour also comes with https://code.soundsoftware.ac.uk/projects/pyin pitch-detection VAMP plugin, which may be preferable for monophonic audio.

In any case regardless of DAW, I think separating the task and using a dedicated Audio to MIDI transformation beforehand is likely the way to go -- and also more Melodyne like :)

Schroedingberg commented 2 years ago

Yes, that is the conclusion I have reached after a night of sleep too. I will probably use my feeble attempts at establishing a workflow (and afterwards an automation for it) to dip my toes into livestreaming. I heard more educational ardour video resources are required, so why not kill two birds with one stone (is there a less violent idiom for that :D ).

Anyway, thank you very much for the consultation/discussion. I guess for any further reports of progress in this matter the ardour forum is probably the better place, isn't it?

Schroedingberg commented 2 years ago

Pyin and especially Tony built upon it looks awesome. Almost a bit like Melodyne: image

Schroedingberg commented 2 years ago

So, currently the action plan is to try and see whether tony solves my specific problem. During trying to get the lua script to run I found something that might be a bug, so I figured I should still share this: LuaException: [string "..."]:6: attempt to call a nil value (method 'nominal_frame_rate') Looks to me like Session:nominal_frame_rate () is no longer defined.