Spatialization And Pitch Issues

[martindevans]

carrotstien has reported two related issues:

• Changing the pitch slider does not change pitch (so therefore dynamic buffering is broken)
• Spatialization is not working unless Spatialize Post Effects is checked on the source
• A null clip (as used by Dissonance when an external spatializater is detected) plays back at the incorrect sample rate

These two items are likely related to changes around how OnAudioFilterRead filters are injected into the pipeline (changed by Unity somewhere between 2017.4 and 2019.1).

Unity 2019.1.2f1

[carrotstien]

my solution was to adjust the rate right in the filter() method: with the call to it from SamplePlaybackComponent being: complete = Filter(session, data, channels, _temp, _temp2, _diagnosticOutput, out arv, out samples, MultiplyBySource, CorrectedPlaybackSpeed);

and _temp2 being identically defined and initialized to _temp

    public static bool Filter(SpeechSession session, [NotNull] float[] output, int channels, [NotNull] float[] temp, [NotNull] float[] resamplingBuffer, [CanBeNull]AudioFileWriter diagnosticOutput, out float arv, out int samplesRead, bool multiply, float rate)
    {
        //Read out data from source (exactly as much as we need for one channel)
        var samplesRequired = output.Length / channels;
        bool complete;
        if (rate == 1)
        {
            //then same as before
            complete = session.Read(new ArraySegment<float>(temp, 0, samplesRequired));
            if (diagnosticOutput != null)
                diagnosticOutput.WriteSamples(new ArraySegment<float>(temp, 0, samplesRequired));
        }
        else
        {
            //change pull more or less samples based on rate, and interpolate with the rate scale to stretch/shrink accordingly
            //without pushing the last frame back into the session pipeline, part of it gets ignored
            //...but because each sample isn't directly tied to a timestamp, 
            //and this is pulling them off of, essentially a stack, this partial sample shift/loss can't be avoided in a simple manner
            var samplesRequiredRateAdjusted = Mathf.CeilToInt(samplesRequired * rate);
            complete = session.Read(new ArraySegment<float>(resamplingBuffer, 0, samplesRequiredRateAdjusted));
            if (diagnosticOutput != null)
                diagnosticOutput.WriteSamples(new ArraySegment<float>(resamplingBuffer, 0, samplesRequiredRateAdjusted));

            int readIndex = 0;
            int writeIndex = 0;
            float readTimeA, readTimeB;
            float writeTime;
            float lerpRatio;
            readTimeA = readIndex / rate;
            readTimeB = (readIndex + 1) / rate;
            //while there is writting to be done...
            while (writeIndex < samplesRequired) {
                writeTime = writeIndex;
                //whenever the write time leads the read time window, shift the read time window until it surrounds the write time
                while (writeTime > readTimeB) {
                    readIndex++;
                    readTimeA = readIndex / rate;
                    readTimeB = (readIndex + 1) / rate;
                }
                lerpRatio = (writeTime - readTimeA) * rate;
                if (readIndex + 1 >= resamplingBuffer.Length) {
                    //this shouldn't really happen, but just in case
                    Debug.LogError("over pulling resample buffer");
                }
                float a = resamplingBuffer[readIndex];
                float b = resamplingBuffer[readIndex + 1];

                temp[writeIndex] = Mathf.Lerp(a, b, lerpRatio);
                writeIndex++;
            }
        }
        //the rest as usual

        float accumulator = 0;

        //Step through samples, stretching them (i.e. play mono input in all output channels)
        var sampleIndex = 0;
        for (var i = 0; i < output.Length; i += channels)
        {
            //Get a single sample from the source data
            var sample = temp[sampleIndex++];

            //Accumulate the sum of the audio signal
            accumulator += Mathf.Abs(sample);

            //Copy data into all channels
            for (var c = 0; c < channels; c++)
            {
                if (multiply)
                    output[i + c] *= sample;
                else
                    output[i + c] = sample;
            }
        }

        arv = accumulator / output.Length;
        samplesRead = samplesRequired;

        return complete;
    }

[martindevans]

An update on this issue: I've been investigating playback in Unity with and without spatializers enabled this week, I've just submitted a PR for testing which makes some changes:

• Removes spatializer support. As far as I can see there is no reliable+low latency way to inject audio into a spatialized audio source
• Applies pitch shifting in Dissonance (Unity have broken their own pitch shifting)
• Uses a new technique for playback which does not rely on null clips (they seem unreliable, particularly in relation to spatial blending).

[martindevans]

Unity have confirmed that they managed to reproduce this issue. Here is the public issue tracker link.

[martindevans]

Dissonance 7.0.2 has just released which includes changes to fix this issue. We've rebuilt the playback system to resample further upstream (as a side effect this helps with audio quality in poor network conditions and improves jitter compensation in all network conditions).

We've also removed support for spatializers as it seems to be impossible to consistently play audio through a spatializer. Some spatializers seem to work and others do not (with no guarantee an updated version of the spatializer plugin won't break things). As noted above, Unity have acknowledged this bug, so I'm hopeful we'll be able to enable spatializer support again in a future version of the Editor.