RPNGOscillator, WavetableSynth, MultiFilter, mutex signaling (feature proposals and arch discussion)

[kyr0]

Hi,

I'm building a fairly complete "pre-wired", polyphonic synthesizer in a classic design. It's basically a clean-room impl. of the design of Xfer Serum [1], but built with Tone.js. I plan to open source it after some code cleanup. I came pretty far so far, including drag & drop based modulation with LFO's and stuff. I will release a preview in a few days for you guys. Here is a little screenshot to give you an idea:

This thingy can also serve as an integration test for Tone.js as I pretty much use any feature in this lib that has to do with synthesis, including all FX. It's probably > 95% code coverage, didn't test it yet however...

Well, to make things really amazing and stunning for you, especially for the first impression, I'd like to introduce a few more features to Tone as well.

However, I don't know how this process goes here... do we have some kind of dev chat where we could discuss architecture of these new features I'd like to work on?

Features I'd like to come up with are:

A) An LFO with a pseudo-number generator involved. Design questions:

• Should it be an additional parameter for the LFO class existing?
• Would you prefer a subclass?

B) User-defined Wavetable synthesis:

• I'd like to drag & drop image and pre-made PCM/wave files, slice them into partials and use it for custom wavetable synthesis
  • 3 modes:
    • PCM blob: User loads a pre-made wavetable (e.g. famous Virus TI ones, etc. other ROM wavetables) -> slice in partials
    • PCM blob mode: FFT transform -> slice in partials
    • Image blob: Light maps to volume (rendering on canvas, -> getImageData() -> histogram -> normalize)
    • MP3 support -> mp3dec (emscripten) -> same process
  • What would be your approach to implement this? My first idea was to extend from the Monophonic instrument class and implement a monophonic wavetable player based synth (WavetableSynth) -- basically like this one [2], but including the additional load and transform implementations.
  • Or should it be a 2-dimensional partials Array? Where a longer part of audio is sliced in mutiple partial arrays? The active partial array is the one played back by the osci -- a wavetable position index points to the partials array that is active and a Signal can trigger the change. This way, an LFO could modulate the partial array to be active and switch it out. That should lead to full control over wavetable playback?

C) MultiFilter

• Basically syntax-sugar, but to have a class that can take an Array of filter configurations would and chains them would be as helpful as the OmniOscillator.

  • I'd like to implement it in a way like EQ3, but with an infinite amount of filters between low and high and freedom of choice for the filter types. Also: Does it make sense to set the rolloff in the MultiBandSplit and make it configurable for each split? Especially for lowpass?

  D) Mutexable Signaling

  • The next big thing is that I have modulators modulating the modulators ;) Also, that I have multiple modulators on one signal. I'd like to implement various algorithms to come up with the "correct next number" for the parameter on a case-by-case basis. Say, for instance: The user sets the cutoff on a filter to 40Hz. Now they add an LFO via a Signal routing. Now I would like to implement a behaviour that relatively adds the 40 Hz + {value of modulator from Signal} Hz per signal change event. How would we implement this? Maybe I'm overlooking an already existing library feature?

  Reading the code I found this zero-ing out connect() function to pass-thru-connect Signals. If this is not existing yet, we could maybe, have a new, optional "via" callback function to be passed on connect(targetNode, /new via callback/ viaFn: (currentValue, signalValue) => nextValue), chain() and fan() {in latter cases would need to support [target, viaFn] syntax per argument}. Then this could be the pure function to be called with the signal parameter and the current parameter and could return the "new" parameter value. By default, it could fallback to "signal parameter" to not change the current behaviour. Wdyt?

  For reference: [1] https://xferrecords.com/products/serum [2] https://github.com/yw662/webaudiofont-ts/blob/master/player.ts#L78

[tambien]

Wow, thanks for the enthusiastic request! I think a lot of these feature requests sound really cool and i would be great to take a look at your proposed implementations. If you do submit some PRs, please break them down into bite-sized chunks.

Here's some of my thoughts:

A) An LFO with a pseudo-number generator involved. Design questions:

I think this would be cool. I could imagine it going directly into the LFO class. I think the API could be something like new LFO(2, 'random').

B) User-defined Wavetable synthesis:

Isn't this currently possible by defining the "partials" attribute of the Tone.Oscillator? It seems like the innovations here that you're describing (such as mp3 and image support) are better placed in the user-facing application that you're building and not necessarily Tone.js since it would require a lot of new code and new APIs (which down the road requires maintaining and cross-browser support, etc).

Though, I wonder if there's an simple API which would support all of these scenarios. Maybe a function which was partialsFromWaveform on the Tone.Oscillator class would do the trick? could be used something like this osc.partials = osc.partialsFromWaveform(waveform: Float32Array): Float32Array

C) MultiFilter

This makes sense to me. I think it'd be pretty simple to implement by chaining multiple filters in a row. I don't think that you'd need the crossover logic like in EQ3. The challenge would be creating an expressive API that would allow you to add/remove/edit all the filters in the MultiFilter.

D) Mutexable Signaling

I think this should be possible with some of the signal math operators currently in Tone.js. You could combine two signals with Tone.Add or Tone.Multiply or Tone.Subtract, as well as a bunch of other operators.

Reading the code I found this zero-ing out connect() function to pass-thru-connect Signals.

Could you link me to the part of the code that references this? i'm not sure i understand what the proposed API is.

[kyr0]

Hey,

thank you for your response! I'm glad you like the ideas and your hints are very helpful.

In the meantime, I already finished "some" way of implementing the "Mutexable Signaling"

The API in my feature branch has a few additions to Signal:

export type SignalConnectionStrategy = "override" | "merge";

export interface SignalOptions {
    ...
    connectionStrategy?: SignalConnectionStrategy; // defaults to "override" to keep the default behaviour
    mergeFrequencyMs?: number; // min 1ms
    mergeSmoothing?: number; // default: 0.8 (Blackman-window) of the samples
    mergeResolution?: number; // min 32 samples (resolution limit)
    getMergeSourcesValue?: (values: Float32Array|Array<Float32Array>, channelCount: number) => number;
    getNextMergeValue?: (values: Array<number>) => number;
}

I have two use-cases in this complex synth:

A: Series of LFO modulation:

LFO1 -> LFO1_mergeSignalAmountNode LFO2 -> LFO1_mergeSignalAmountNode

Therefore, the LFO modulates the other LFO which leads to an inherent polyrhythm later (especially because both LFO's have different frequency and min/max values).

LFO1_mergeSignalAmountNode uses default merge impl: Average of both signal data streams makes up the new output value stream.

B: Result of series of LFO modulation modulates Master Filter cutoff (relatively to base frequency)

Master filter cutoff frequency -> MasterFilter_MergeSignal LFO1_mergeSignalAmountNode -> MasterFilter_MergeSignal [...] whatever else signals -> MasterFilter_MergeSignal

Here, the resulting cutoff frequency is determined by the user defined cutoff frequency + the (maybe many) modulations:

    this.amountNode = new Tone.Signal({
            connectionStrategy: 'merge',
            getNextMergeValue(values: Array<number>) {
                // first value in array is the first connected signal, which is the user defined cutoff (static)
                const userDefinedCutoff = values[0];
                let sumOfCutoffModulations = 0;
                // all other signals are modulations added dynamically via drag&drop (modulation matrix)
                if (values.length > 1) {
                    for (let i = 1; i < values.length; i++) {
                        sumOfCutoffModulations += values[i];
                    }
                }
                return userDefinedCutoff + sumOfCutoffModulations;
            },
        });

The problem with my solution here is, that I wasn't aware of the Add/Multiply Tone nodes before you mentioned them. Now I got the black magic with the ConstantToneSource and Gain behind the scenes. Pretty smart... and I guess it's quite performant using only native Web Audio API's. What I did is much more flexible because it doesn't use side-effects.. but it's limited in resolution and performance.

Actually, I guess I could probably also go with Tone.Add now xD Maybe I try this in a few days but yeah, last night, after poking around on my own and I didn't came to the point of a working solution on my own with native API's only in like 1-2h... I came up with a Timer/Tone.Analyzer approach.

As I said, it works great and sounds amazing. But it is limited to a resolution of 32 samples, numbers only and 1000 merge/sync ops per second. Also, it is a timer that doesn't run in a worklet or something atm. So I'm afraid of performance issues if used extensively. However processing is smooth as my impl. uses WeakMap, a lot of caching and manages resources of course. Yeah..., it works great for my use-cases atm - to my very surprise :))

I can of course PR it, so you can have a look anyways.

Now I'll work on the other features.

Did you think of a Pitch-control for the NoiseSynth or Noise Source? Serum calls it "Key tracking". I really think it's just a note -> hz mapping and a lowpass filter :)

Also, for the noise source, we could introduce some "randomness" factor. This one could just re-initialize the pre-generated noise buffer at times, to get a better "randomness" feeling. Also "randomness" could have a modulation effect on the lowpass filter (fractions) if we have that...

Of course, all my PR's will be fresh from "develop" and only contain certain features on those feature branches 👍

And happy new year to you and your family!!!

It's an amazing project - thank you for all the work!! 🥇 💯

I can't wait to show this synth to you soon as a demo. It already sounds amazing. A friend of mine will sound design some presets as well :)

ps.: I might also come up with a ModalSynth for modal synthesis and more physical modeling. The PluckSynth is already nice but could also be improved a little bit further :) Well, I just took this online course from Stanford University [1] and it seems quite doable to implement the algorithms necessary to also have a modalSynthesisModelFromWaveform alongside partialsFromWaveform -- I know FFT quite well from a past project so I guess I could come up with an algo for auto-correlating the dominant frequencies and amplitudes of the harmonic series of input PCM data to make a nice additive sine + envelope based physicial modeling synth...

[1] https://online.stanford.edu/courses/sohs-music0002-physics-based-sound-synthesis-games-and-interactive-systems

[kyr0]

And yeah, after looking deeper into the code base etc. I came to the conclusion that the partials already do exactly what I wished for (sry for the lot prosa about this in my initial text) -- only the dimension of sliding between different waveforms would be still missing in contrast to advanced wavetable synths like Serum.

To give you an idea of what I mean.. this is just a saw I could easily come up with using the partials of course:

But here you can see that the synth also has a "WT pos" - a wavetable position. This means that the OSC stores an Array<Array< Partial >> if that makes sense and will set the partials for the oscillator to use at the time via the WT pos parameter (the 0 in the right bottom corner indicates wave table 0 of idk 256 here):

Now you can modulate that WT pos parameter with an LFO and guess what, you'll get a very very decent sound at times :)

I think this would be a great addition to the functionality because with that it would be possible to use the proposed partialsFromWaveform in a row... imagine -- we have a sound of 3 seconds; chunk it, call partialsFromWaveform per chunk and make this array of partials-wavetable out of it. Let's say, hard limit to 64 arrays of partials. You can then define the playback speed via an LFO modulation (and limit the section of playback via min/max value of course) and the sound will morph through the ever-slightly changing waveforms. Like a pulse-width modulation but with any waveform you can think of.

[kyr0]

One more simple thing: The LFO seem to not pass partials down to the Oscillator as an option. Would be cool to have custom waveform for LFO's too :) Then we're in professional synth land soon :)

[kyr0]

@tambien Wow, cool! Thank you that you did work on the LFO partials support already. I'm working on the UI for the partials and got a nice thing working over here. Now I can add this for the LFOs as well. Great :)

I'm back to my daily business job since a week and have less time now, but really can't wait to publish the first alpha :)

[kyr0]

@tambien Actually the idea with the "random" LFO is now possible! :) Because with the partials in LFO, we can generate a random partials array and even flip it regularly... amazing! This allows for true chaotic modulation which can be a really nice way to come up with special sound designs

@tambien Btw, I dropped my impl. for Merge Signals and used the Add Modifier Signal as you suggested. That's why I didn't come up with a PR yet. I want to implement all Tone.js features with the UI first. Then I will come up with PR's to implement those features suggested soon after.