Signals of audio frames of an unknown size (number of channels)

[mitchmindtree]

I'm currently writing an audio engine for an exhibition that will run over 100 channels of audio from a single audio server. Much of the audio content played throughout the exhibition will have different numbers of channels. Currently, there isn't really a nice way of working with signals of audio frames in the case where the number of channels is unknown or may vary widely.

It would be nice if the sample crate came with some abstraction for working with these kinds of signals that felt as nice as the Signal trait itself. That said, some of the functionality will inherently be more costly and limited due to the need for comparing numbers of channels at runtime rather than compile time. Some form of dependent types (or maybe integer generics?) could possibly keep this type-safe, but it looks like neither of those will land for a while at least.

I can imagine a trait that might look something like this:

// A signal of interleaved audio samples.
trait InterleavedSamples {
    // The type of samples yielded by the signal.
    type Sample: Sample;
    // An iterator yielding each sample for a single frame.
    type Frame: Iterator<Item=Self::Sample>;
    // The number of channels per frame.
    fn channels(&self) -> usize;
    // Yield the next frame as an iterator.
    fn next(&mut self) -> Self::Frame;

    // Provided methods:
    // - map
    // - zip_map
    // - offset_amp
    // - scale_amp
    // - mul_hz
    // - from_hz_to_hz
    // - scale_hz
    // - delay
    // - clip_amp
    // - inspect
    // - bus
    // - take
    // - by_ref
}

It would be a pity to require this much duplication of the Signal trait... It would be great if we could still somehow use the Signal trait. Perhaps it's possible by revisiting the Frame trait and splitting it into fixed-size and dynamically-sized functionality?

[MOZGIII]

This is still relevant for me. It feels like the Frame type itself should not be specified over the channels amount. This wouldn't be a problem if the Signal type wasn't relying on the frame.

I'd propose reviewing the sample crate implementation to find all the places where the Frame::NumChannels is actually used. In addition, I invite everyone who sees this to post some use cases for the Frame and Signal types - to see if we actually need Frame to specify NumChannels at a type level. Note in mind that Frame is (currently) not Sized - so it doesn't have to require a size known upfront from an implementation type.

[MOZGIII]

I did some experiments, and it definitely looks like the Frame trait that we have currently is too tightly coupled with the implementations. Frame with dynamic channel amounts (unknown at compile time) is not allowed by the current trait design, so it's not even possible to implement in the current version.

To sum up:

Perhaps it's possible by revisiting the Frame trait and splitting it into fixed-size and dynamically-sized functionality?

+1 on this, definitely the way to go.

[MOZGIII]

I created #114. Also, I'm working on some code in my project to solve this issue, I may submit a PR to sample here once I get it done.

[MOZGIII]

I suffered quite significant difficulties with using Frame type during my experiments, mainly because Frame type is expected to be owning, not referencing Samples. This seems like a major concept: turn frames into mut-borrows instead of owners, and only allow frame operations to be performed in-place. What do you think about this idea, @mitchmindtree ?

[mitchmindtree]

Thanks for digging into all this @MOZGIII! I'm also still interested in tackling this. I'm currently overseas spending some time with family but will hopefully have some time to revisit these issues and respond to you properly in ~ a week.

[MOZGIII]

Cool! I'd be happy to share my results with you, and discuss how to fix the usability of the crate. I've been stuck with this for about a week now, and I'm mostly poking the compiler around with various attempts to fix either the usability of the current design, or the very design of the Frame type itself. Turns out it's not an easy problem to solve at all.

[jneem]

+1 on allowing Frame to have dynamic size.

When looking in how to use dasp_signal for processing wav files, I noticed that the resample example is buggy: it pretends to interleave samples from the input, but actually the number of channels is fixed at 1. So if the input is, say, stereo then it's doing the wrong thing.