search

performous / composer

Song editor for Performous and other singing games
Other
39 stars 23 forks source link

displaying spectrogram in the background #38

Open juergendrachta opened 4 years ago

juergendrachta commented 4 years ago

I'd find it very beneficial to be able to display the calculated spectrogram in the background. It definitely would help adjusting the pitches, when the automated pitch extraction fails.

As an alternative, I'd propose to be able to set an own spectrogram as a background (e.g. I separate the vocals from the songs via Spleeter, and then draw a spectrogram via python's scipy/matplotlib). Of course, in this case we have to agree on a given format, i.e. how many Hz or ms a pixel represents in the image.

Tronic commented 4 years ago

Doesn't the composer already show you a spectrogram? It could, however, use the improved algorithm I made for Performous a few months ago. The old one is noisier. Standard spectrogram (what you see in music players and other such software) is way too low resolution. You need to do FFT reassignment method and use a sufficiently short FFT window (1024 samples @ 48 kHz seems optimal) to get clear results. Harmonics combining (i.e. the pitch detection we have) is also useful for removing unnecessary clutter but I agree that in case of composer it might be sometimes useful to omit that step because occasionally the correct singing frequency gets hidden as a harmonic of another lower frequency instrument (in this case showing a line exactly one octave lower).

Tronic commented 4 years ago

That being said, I have what you need implemented in Python so that it paints into an image (Numpy array). It is a bit slow but probably not too much so for composer. Having someone implement this is another thing as AFAIK no-one is working on Composer at this time, and I am not interested in touching the Qt code either. If you are willing to make a PR, I'll help with the algorithmic details.

juergendrachta commented 4 years ago

Thanks for the response!

I looked through all the menus composer offers, but I only see the option to display the extracted notes from spectrogram and not the spectrogram itself. I looked a bit into the code, and it seems to me, that it's not a one-liner to display the complete spectrogram instead of the detected pitches. Somehow it's understandable to directly draw the extracted notes via some Qt's drawline methods, because than you don't have to think aliasing and re-scaling issues of the spectrogram's pixel graphic.

Because of those issues, I would find it nice, to rather implement a functionality, which allows the user to load a self-built spectrogram in the background, because than the decision if left to the user, which resolution is taken and which audio file is used as basis. Furthermore, I think that is easier to implement.

And about Qt: For me it's quite the same! :-) I'd try to avoid as much as Qt possible - the main reason is, that i almost know no Qt, especially Qt in C++.

And about the FFT and spectrogram building on its own: Where can I find the code you are using for building the spectrogram? Because maybe I simplify my "workflow", which consists now of

This is how my spectrograms normally look like (generated via plt.specgram(monowav, Fs=FS, NFFT=10*1024, noverlap=8*1024, scale="dB", cmap="gist_ncar") # plot), which would like to be loaded in the composer :-D image

And thanks for the offer to help in a PR, I'll see how far I can get into the composer source code in my spare time. If I see, that it's easier than expected for a Qt-newbie as me to get a heatmap of a numpy array correctly drawn and aligned correctly with the time and frequency axis, then I'll come back to your offer! (y) :-)

Tronic commented 4 years ago

I'll get you some Python code tomorrow. C++ implementation is available in https://github.com/performous/performous/blob/d962d38acaec895b497029e5909ce9b5b9d7149c/game/pitch.cc#L107

Tronic commented 4 years ago

Sorry for the delay. I got suddenly flooded with work but will try to get back to in a couple of weeks if you are still interested.

MikeEA0 commented 2 years ago

Any progress on this one?

This seems to be a rather useful feature to me, since in the song I am currently trying to time, very often pitches of the instruments are shown rather those of the vocal...

Tronic commented 2 years ago

No progress on my part, and unfortunately I won't have time to work on it on foreseeable future. Hoping that someone else could pick this up.