How to set Frame Size?

[sainteckes]

I want to set a specific frame size of the audio callback. Is that possible in cubeb? Right now I always get 512 samples. Thanks you for your support!

[padenot]

Change the latency parameter. What you'll get is very dependent on the backend, unfortunately. The name is kind of misleading, generally it really is the buffer size (in frames), but it definitely impacts the latency.

And to be specific, in general the frame size is the size of one audio frame, and an audio frame is all the samples that happen at the same time (one per channel). The size of the frame is therefore the number of channel (if in samples) or sizeof(sample_type) * channel_count in bytes. But I'm pretty sure you're talking about the buffer size here. A drawing that hopefully makes everything clear: https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/Basic_concepts_behind_Web_Audio_API#audio_buffers_frames_samples_and_channels (written in the context of the Web Audio API, but it's the same concepts).

cubeb uses those terms rather specifically.

You can't set a very specific frame size in general (for all backends and hardware), because cubeb doesn't generally perform callback adaptation, this is best done at the application layer (e.g. Firefox does more or less this). The goal is to get a "good" buffer size, to maximize performance -- users can later decide what to do if they need something else (some background I wrote some time ago, but it's the same problem: https://github.com/WebAudio/web-audio-api/issues/2450#issuecomment-572469654).

You can query the minimum latency supported and check the actual latency you have (roundtrip by summing or one way for input or output) using three other methods.

[sainteckes]

Wow that was really educational, I did some Plugin Development but since I want to build audio apps in Rust instead of Juce I have to dig deeper into audio I/O. Right now I am still using the Juce audio callback that I wrapped in Rust, but it is way too heavy to build for what it is doing (not to mention the licensing).

Yes it is the buffer size or number of frames what I was talking about. With the drawing I will never mix this up again :) The Callback adaption paper is super interesting, and I might implement it for my app. So the method in a paper (i guess implemented with a ringbuffer) is what you would recommend when my processor needs to do an FFT with a fixed size? And in general super amazing what you are doing with web audio, I hope real DAWs and so on are working in the browser soon :)

[padenot]

So the method in a paper (i guess implemented with a ringbuffer) is what you would recommend when my processor needs to do an FFT with a fixed size?

Yes, or get an FFT implementation that can do non-power-of-two transforms. Off the top of my head, pffft and tx from ffmpeg are both fast and support power of two size and a factor that can be 3 or 5 (e.g. 3 * 64 = 192 for example, which is a common size on Android).

If this works for you then it's perfect, if it doesn't, ring buffer it is (or you can also do tricks if you in fact want to do a convolution, https://fgiesen.wordpress.com/2023/03/19/notes-on-ffts-for-users/ is a good read).

[sainteckes]

Yes good read! I want to feed a neural network that is only trained on a spectrogram of a specific size, so currently it needs a fixed size input that is used to calculate an STFT. Additional latency is not perfect because the model is already adding half a buffer size. In the fft document ryg is saying that "arbitrary-size FFT algorithms work by reducing a size-N FFT to a different FFT of a nearby “convenient” size, again with pre-/post-processing." I don't know what this post processing involves, but probably some sort of interpolation? Could this help to support different frame sizes as well?

[sainteckes]

Btw when I call stream.input_latency() (I call it right after starting the stream), it shows the Error Message Error { code: NotSupported }. stream.latency() is returning zero... I am running on macOS 13.2.1.

[padenot]

On macOS you can only retrieve this after the stream has been running for some time, because (part of) the number comes from something the real-time audio callback passes as argument.

But also it sounds like you might be using the C++ backend for macOS, you should be using the rust backend, it's probably something we should be offering as a feature or something. Maybe it's around here? https://github.com/mozilla/cubeb-rs/blob/master/cubeb-sys/build.rs

[padenot]

In C cubeb you just clone the rust backend (https://github.com/mozilla/cubeb-coreaudio-rs/) in the src/ dir and ask the build system to build the rust backend, I think we need to propagate this to the rust layer above.

[sainteckes]

it is defnitely running cmake and building cubeb_audiounit.cpp and cubeb_osx_run_loop.cpp. I don't see a possibility to specift something related to the macos backend in the build.rs file. But I can try to add the flags to the build.rs specified here: https://github.com/mozilla/cubeb-coreaudio-rs/blob/trailblazer/build-audiounit-rust-in-cubeb.sh

[padenot]

Yes, sorry about that, we're using another build system in Firefox so we're missed updating this one.

[sainteckes]

I made it download the repo and use the cmake command, but then it complains because of a dependency problem..

I might do a PR for this buils.rs extension and we can continue fixing this there?

Execution failed (exit code 101).
/Users/aicse/.cargo/bin/cargo metadata --verbose --format-version 1 --all-features --filter-platform aarch64-apple-darwin
stdout :     Updating crates.io index
error: failed to select a version for `cubeb-sys`.
    ... required by package `cubeb-core v0.10.3`
    ... which satisfies dependency `cubeb-core = "^0.10.3"` of package `cubeb-backend v0.10.3`
    ... which satisfies dependency `cubeb-backend = "^0.10.3"` of package `cubeb-coreaudio v0.1.0 (/Users/aicse/Development/cubeb-rs/cubeb-sys/libcubeb/src/cubeb-coreaudio-rs)`
versions that meet the requirements `^0.10.3` are: 0.10.3

the package `cubeb-sys` links to the native library `cubeb`, but it conflicts with a previous package which links to `cubeb` as well:
package `cubeb-sys v0.10.3 (/Users/aicse/Development/cubeb-rs/cubeb-sys)`
    ... which satisfies path dependency `cubeb-sys` (locked to 0.10.3) of package `cubeb-core v0.10.3 (/Users/aicse/Development/cubeb-rs/cubeb-core)`
    ... which satisfies path dependency `cubeb-core` (locked to 0.10.3) of package `cubeb v0.10.3 (/Users/aicse/Development/cubeb-rs/cubeb-api)`
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the links ='cubeb-sys' value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

failed to select a version for `cubeb-sys` which could resolve this conflict

stderr : 

[padenot]

Sure, thanks!

[sainteckes]

For the record: https://github.com/mozilla/cubeb-rs/pull/82