initial playback latency

[mgood7123]

is there any way to decrease the initial playback latency without underrunning?

as at the moment I have

        put(SAMPLE_RATE, 44100);
        put(BUFFER_SIZE_BYTES, get(SAMPLE_RATE));
        put(CONTROL_THREAD_POLLING_MILLIS, 5);
        put(MIN_MILLIS_AHEAD_TO_WRITE, 400);
        put(MIN_WRITABLE_CONTENT_NANO, 5000);
        put(TIMESTAMP_POLLING_DELAY_MILLIS, 50);

in the config

and I don't know how to reduce initial playback latency without underrunning

and I am not as experienced with the internals of PFSeq to be able to make it into a real-time audio streamer without affecting the perfect timing of PFSeq

eg where any audio written will immediately be played

in particular, for use with a MIDI sequencer, where a NOTE can be played at any time, and is NOT limited to the constraints of a piano roll (eg it could be located at a position which does not perfectly align with a beat grid of 1/4 or 1/3 or whatever the user desires)

also the same issue would occur in a midi sequencer (eg drum pattern), where a user IS NOT guaranteed to press the play button 100% in sync with the metronome, but EXPECTS the audio to immediately play with the sync adjusted to match when ever the play button has been pressed

eg if the user presses play on a triplet, then the sync should be adjusted such that every 4 steps is on a triplet eg

WHAT THE USER EXPECTS:
       beginning of bar                                                        end of bar
       |                                                                       |
       beginning of 1st beat               beginning of 2nd beat               |
       |                                   |                                   |
       -------------------------------------------------------------------------
       |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |
       |  :  :  |  :  :  |  :  :  |  : Pos |  :  :  |  :  :  |  :  :  |  :  :  |
       |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |
       -------------------------------------------------------------------------

THE ACTUAL GRID
       beginning of bar                                                        end of bar
       |                                                                       |
       beginning of 1st beat               beginning of 2nd beat               |
       |                                   |                                   |
       |     beginning of bar              |                                   |     end of bar
       |     |                             |                                   |     |
       |     beginning of 1st beat         |     beginning of 2nd beat         |     |
       |     |                             |     |                             |     |
       -------------------------------------------------------------------------------
       |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :
       |  : PLAY|  :  :  |  :  :  |  : Pos |  :  :  |  :  :  |  :  :  |  :  :  |  :  :
       |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :  |  :  :
       -------------------------------------------------------------------------------

[peoplesfeelings]

Good question.

The AudioTrack class, from the Android SDK, comes with the limitation that data cannot be cleared from the buffer, once it is written to the buffer, unless you stop the AudioTrack. So it may be unavoidable that some amount of latency will exist, though it may be possible to reduce it to much less than 400 ms.

The way to try to lower the latency would be lowering MIN_MILLIS_AHEAD_TO_WRITE. If you lower it too low, something will break. Adjusting other config values, or editing the module in some way, might accomodate a significantly lower MIN_MILLIS_AHEAD_TO_WRITE. I played around with it a bit. I would guess it wouldn't be safe to go much lower than 200ms, at least safely on any Android device.

There is another thing to take into account.

Because AudioTrack does not allow clearing the buffer, while playing, and because near-instantaneous changes were not a design requirement, originally, PFSeq writes clips in their entirety to the buffer. Like, with the metronome, the entire tick audio clip is written, and nothing else can be written to that AudioTrack, until that tick has completed. So, if you wanted something approaching instantaneous change of output, then we would need to update the module to split up pending audio clips into smaller chunks. That might not be a bad idea. There could be a PFSeqConfig setting for audio clip chunk size, which probably couldn't get too small. I think pushing expensive operations to the existing track work threads would be asking for trouble, because they need to keep up with the buffer writes. Even at 400ms latency, I wonder if any devices would not get data written in time to avoid an underrun. We could code a new work thread for each PFSeqTrack instance, to handle the chopping. I was thinking about doing one of those threads, anyway, so that the necessary fade-out operations could be done outside of the PFSeq control thread. I'm not sure at what point the number of threads would be problematic, at least on some devices.

The speed of buffer writes is outside the control of any Android code, I think. AudioTrack interfaces with the "native layer," underlying codecs and firmware (or something), specific to the devices, and outside the control of the Android system. Each device has an inherent amount of time that is required for a buffer write to complete, and we have to consider the maximum time possible in any scenario. That's one of the reasons for the 400ms latency.

Worth exploring, though.

Another way to break up clips would be for the app that is implementing PFSeq to break up its audio clips into small sections.

I can point you to the following, which you may be aware of.

• article on the People's Feelings site about the module, providing an overview of the design. http://peoplesfeelings.com/pf-seq-a-precision-timing-audio-sequencer-for-android/
• the comments at the top of PFSeqConfig, which describe the settings
• the definition of the contentWriting Runnable, in PFSeq, which is the guts of the engine

Hope that helps.

[peoplesfeelings]

I just saw the second half of your message, which I think you added in an edit.

In light of the technical limitations inherent in the Android system, as I understand it, which I describe above, I think there may still be a way for you to accomplish your use cases, using the PFSeq module, in a reasonable way.

At the level of the app that is implementing PFSeq, the real-time audio note could be played from outside of PFSeq, with your own AudioTrack instance(s) in static (non-streaming) mode. On subsequent playback or loop iterations, the audio could come from PFSeq, assuming that the data is also set in the PFSeqTrack.pianoRoll.

You mention timing that is apart from grid snap positions. This can be accomplished with PFSeqTimeOffset instances set to MODE_PERCENT. See the comments at the top of the PFSeqTimeOffset class for more info on that.

Regarding the last use case that you mentioned, in light of the inherent technical obstacle that I described, one solution is to just put the data into PFSeq, and the user just won't hear the first audio clip(s) that start within the first 400ms, or whatever MIN_MILLIS_AHEAD_TO_WRITE is set to.

Alternately, those first clips from the first 400ms of the triggered beat could be played by the implementing app, with less-precise timing. The implementing app could utilize the PFSeqPianoRollItem.timeOffset instances to recreate the clip play time targets, by recreating the calculations found in PFSeqPianoRollItem.soonestNanoAfter(nano). To give you an idea, that method's return value is: currentBeatNanotime + (long) (beatsOut * nanosPerBeat) + offsetFromBeatNano

You can call the relevant functions from PFSeq, which are public. When you know the target play times for those first clips, you can play them with your own static-mode AudioTrack instances (or MediaPlayer or SoundPool, they all exhibit similar imprecision), using some timer function. The clip play time target values (nanotime) correspond to the Java System.nanoTime() clock.

The user may notice a timing irregularity at the outset of the triggered beat, due to the imprecision/lag of triggering clips programmatically, but the user should intuit the cause as a technical limitation, and recognize the precision timing that follows. And they would recognize that subsequent playback or loop iterations have the precise timing.

[mgood7123]

I just saw the second half of your message, which I think you added in an edit.

In light of the technical limitations inherent in the Android system, as I understand it, which I describe above, I think there may still be a way for you to accomplish your use cases, using the PFSeq module, in a reasonable way.

At the level of the app that is implementing PFSeq, the real-time audio note could be played from outside of PFSeq, with your own AudioTrack instance(s) in static (non-streaming) mode. On subsequent playback or loop iterations, the audio could come from PFSeq, assuming that the data is also set in the PFSeqTrack.pianoRoll.

You mention timing that is apart from grid snap positions. This can be accomplished with PFSeqTimeOffset instances set to MODE_PERCENT. See the comments at the top of the PFSeqTimeOffset class for more info on that.

Regarding the last use case that you mentioned, in light of the inherent technical obstacle that I described, one solution is to just put the data into PFSeq, and the user just won't hear the first audio clip(s) that start within the first 400ms, or whatever MIN_MILLIS_AHEAD_TO_WRITE is set to.

Alternately, those first clips from the first 400ms of the triggered beat could be played by the implementing app, with less-precise timing. The implementing app could utilize the PFSeqPianoRollItem.timeOffset instances to recreate the clip play time targets, by recreating the calculations found in PFSeqPianoRollItem.soonestNanoAfter(nano). To give you an idea, that method's return value is: currentBeatNanotime + (long) (beatsOut * nanosPerBeat) + offsetFromBeatNano

You can call the relevant functions from PFSeq, which are public. When you know the target play times for those first clips, you can play them with your own static-mode AudioTrack instances (or MediaPlayer or SoundPool, they all exhibit similar imprecision), using some timer function. The clip play time target values (nanotime) correspond to the Java System.nanoTime() clock.

The user may notice a timing irregularity at the outset of the triggered beat, due to the imprecision/lag of triggering clips programmatically, but the user should intuit the cause as a technical limitation, and recognize the precision timing that follows. And they would recognize that subsequent playback or loop iterations have the precise timing.

hmmmm ok, sounds complex tho

[peoplesfeelings]

As I understand the limitations, after spending time researching the problem, I think this is the trade-off of the absolute precision. You need to have the tracks running, and the track buffer must have data written at least up to the current time, and that buffer data cannot be cleared. The question is how far ahead to write. In a controlled scenario, this could be very low, but underruns must be avoide at all costs, and a lot goes on in that central engine (the contentWriting Runnable): it's managing multiple tracks, allocating and generating arrays of silence, and it has to chop clips to size and apply fade-outs to them, to avoid clipping. And we have to worry about cheap devices. So there must be some delay after input. Right now the default MIN_MILLIS_AHEAD_TO_WRITE is 400ms, and I think it's possible to get that lower. I made it configurable so developers can make the choice, themselves. I opted on the safe side, with that default. If you set it to 200 then you might find it runs fine on your device. To a user, they would hardly notice if they needed to anticipate the drop by 200ms in order to hear it audibly, and it would seem like an expected technical limitation, maybe. So, maybe you could actually include an artificial delay, for that live commad, which matches the MIN_MILLIS_AHEAD_TO_WRITE. Schedule the beat the user triggers a bit in the future, the same distance, or maybe a little more. The user will learn to anticipate by that amount.

Also, I think it would not be an unreasonable pattern to set MIN_MILLIS_AHEAD_TO_WRITE low, and then use a handler for underrun errors, which then changes the config value to a higher value, and calls the setup method, to reconfigure the sequencer, and maybe display a message that the sequencer configuration was updated, due to device limitations. For a value of 200, this may not happen in many scenarios. I just recall playing with that and arriving at 400, thinking it was fairly safe.

[mgood7123]

if it helps, in AAudio ( https://developer.android.com/ndk/guides/audio/aaudio/aaudio ), I get numFrames 192, bufferSize 384, bufferCapacity 384

/**
 * Set the requested data callback buffer size in frames.
 * See {@link #AAudioStream_dataCallback}.
 *
 * The default, if you do not call this function, is {@link #AAUDIO_UNSPECIFIED}.
 *
 * For the lowest possible latency, do not call this function. AAudio will then
 * call the dataProc callback function with whatever size is optimal.
 * That size may vary from one callback to another.
 *
 * Only use this function if the application requires a specific number of frames for processing.
 * The application might, for example, be using an FFT that requires
 * a specific power-of-two sized buffer.
 *
 * AAudio may need to add additional buffering in order to adapt between the internal
 * buffer size and the requested buffer size.
 *
 * If you do call this function then the requested size should be less than
 * half the buffer capacity, to allow double buffering.
 *
 * @param builder reference provided by AAudio_createStreamBuilder()
 * @param numFrames the desired buffer size in frames or {@link #AAUDIO_UNSPECIFIED}
 */

[mgood7123]

so I did some more research on IRC, and to support a tempo grid

eg

        // A timebase that allows sequencing in relation to musical events like beats or bars
        // it may look something like this:
        // data[4] = {MIDI_ON, MIDI_OFF, MIDI_ON, MIDI_OFF}; ?
        // with the data being mapped to a grid of 1/4 notes
        // bar[0] = {data[0], data[1], data[2], data[3]};
        // 120 bpm == 30 bars x 4 beats x 1/4 => 0.5 1/4 notes per second if I am right.
        // That means 1/4 note is 0.5 × 48k samples long
        // ( 4/4 tempo )
        // data[0] is frame 0, data[1] is frame 24000, data[2] is frame 48000, data[3] is frame 72000
        // bar[0] is frame 0, bar[1] is frame 96000, and so on
        // tempo_grid[4] = {0, 24000, 48000, 72000}; in 1/4 notes, assuming 48k sample rate and a bpm of 120
        // when each note is aligned to 1/4 notes
        // eg note quantisation, (snap to resolution, eg snap to 1/4)

would PFSeq need to be... partially redesigned?

[mgood7123]

if it helps further: I managed to create a tempo synced metronome in AAudio, which appears to stay perfectly in sync with PFSeq's metronome (not sure if buffer underrun's would cause it to become desynced)

also the latency is about 4ms (48000 sample rate)

I/chatty: uid=10241(smallville7123.aaudiotrack) AudioTrack identical 18 lines
W/AudioEngine: writing 4 milliseconds (192 samples) of data
W/AudioEngine: writing 2.66667 milliseconds (128 samples) of data
    writing 1.33333 milliseconds (64 samples) of data
W/AudioEngine: writing 4 milliseconds (192 samples) of data
I/chatty: uid=10241(smallville7123.aaudiotrack) AudioTrack identical 8 lines
W/AudioEngine: writing 4 milliseconds (192 samples) of data
W/AudioEngine: writing 1.33333 milliseconds (64 samples) of data
    writing 2.66667 milliseconds (128 samples) of data
W/AudioEngine: writing 4 milliseconds (192 samples) of data

https://github.com/mgood7123/AAudioTrack/blob/master/AAudioTrack2/src/main/cpp/ardour/AudioEngine/AudioEngine.cpp#L387

[peoplesfeelings]

cool, thanks

[mgood7123]

cool, thanks

updates since last post:

fixed the popping from my previous commit (I was not outputting silence when there are no events and the engine is playing) - https://github.com/mgood7123/AAudioTrack/commit/c81c4166eff89d9694a6b4034693b8833e97d42c

fixed an issue where the buffer size would be set to zero when an underrun occurs - https://github.com/mgood7123/AAudioTrack/commit/75c2f4f0a567724616e85d09151ad1ddfe047c99

https://github.com/mgood7123/AAudioTrack/blob/master/AAudioTrack2/src/main/cpp/ardour/AudioEngine/AudioEngine.cpp#L387

https://github.com/mgood7123/AAudioTrack/blob/master/AAudioTrack2/src/main/cpp/ardour/Backends/AAudio.cpp

[mgood7123]

cool, thanks

also https://github.com/android/ndk/issues/1383

[peoplesfeelings]

Awesome, I appreciate it. I'm bogged down on another project, and have other stuff slated for next, but am definitely interested to return to this. The NDK stuff will be great to explore. If anyone wants to submit PRs on feature branches, I would definitely review them. The discussion in this issue and the references I linked should provide a good overview of the current state of the project, and the directions it could move in.

Also am interested to check your projects, it seems you know about this stuff. Cheers.

[mgood7123]

Awesome, I appreciate it. I'm bogged down on another project, and have other stuff slated for next, but am definitely interested to return to this. The NDK stuff will be great to explore. If anyone wants to submit PRs on feature branches, I would definitely review them. The discussion in this issue and the references I linked should provide a good overview of the current state of the project, and the directions it could move in.

Also am interested to check your projects, it seems you know about this stuff. Cheers.

haha I probably don't know as much as you :)

[mgood7123]

Awesome, I appreciate it. I'm bogged down on another project, and have other stuff slated for next, but am definitely interested to return to this. The NDK stuff will be great to explore. If anyone wants to submit PRs on feature branches, I would definitely review them. The discussion in this issue and the references I linked should provide a good overview of the current state of the project, and the directions it could move in.

Also am interested to check your projects, it seems you know about this stuff. Cheers.

added a Mixer :)

[mgood7123]

got a working, basic sampler, with a right panned delay (not echo) plugin :)

https://github.com/mgood7123/AAudioTrack/blob/master/AAudioTrack2/src/main/cpp/ardour/AudioEngine/AudioEngine.cpp#L408

[mgood7123]

also, on regards to tempo timing

        // the sample counter is used to synchronise events with frames
        // A timebase that allows sequencing in relation to musical events like beats or bars
        // it may look something like this:
        // data[4] = {MIDI_ON, MIDI_OFF, MIDI_ON, MIDI_OFF};
        // with the data being mapped to a grid of 1/4 notes
        //
        // see TempoGrid for info on tempo mapping
        //
        // data[0] is frame 0, data[1] is frame 24000, data[2] is frame 48000, data[3] is frame 72000
        // bar[0] is frame 0, bar[1] is frame 96000, and so on
        // tempo_grid[4] = {0, 24000, 48000, 72000}; in 1/4 notes, assuming 48k sample rate, 120 bpm
        // when each note is aligned to 1/4 notes
        // eg note quantisation, (snap to resolution, eg snap to 1/4)

        // If the tempo grid is set up to 120 bpm with 4 notes per bar,
        // on each note, if there is an event associated with that note,
        // the associated generators will trigger for that event.
        // For example, a Sampler and a Synth are assigned to play every
        // 1st note of every bar, when the engine reaches this note,
        // it should call the processing callbacks of the Sampler,
        // and the Synth, with their output each in a seperate mixer
        // input port

https://github.com/mgood7123/AAudioTrack/blob/master/AAudioTrack2/src/main/cpp/smallville7123/TempoGrid.h https://github.com/mgood7123/AAudioTrack/blob/master/AAudioTrack2/src/main/cpp/smallville7123/TempoGrid.cpp https://github.com/mgood7123/AAudioTrack/blob/master/AAudioTrack2/src/main/cpp/ardour/AudioEngine/AudioEngine.cpp#L466

[peoplesfeelings]

Hm, very cool. Will be fun to dive back into the Android audio stuff, with C++, especially. I think you may be providing me with some good leads with which to redo the PFSeq idea with much less latency.

What's your take on the current state of the Android app market, regarding these kinds of real-time music composition tools? Are there particular tool sets or developers who I definitely need to check out? Who has the most solid "engine"?

Nice work on your AAudioTrack project. Haven't dug in, yet. Allocating my software juice on another project I need to get done. Looking forward to it, though.

[mgood7123]

I managed to get my Audio engine to play 120 audio tracks simultaneously (at the same time) with rare under-runs by using a 8k buffer, however a 192 buffer works fine for up to 60 audio tracks playing simultaneously

also improved stuff, fixed important bugs, and cleaned up code a bit :)

https://github.com/mgood7123/AAudioTrack/commits/master

though it seems to... waver out of sync even with just 1 audio track playing, like it is in sync then it goes a bit out of sync then it caches up to sync then it goes a bit out of sync again and then it catches up and so on, however it does seems to stay in sync, when it goes out of sync it goes back into sync again

I do not know why it temporarily goes out of sync

[mgood7123]

Hm, very cool. Will be fun to dive back into the Android audio stuff, with C++, especially. I think you may be providing me with some good leads with which to redo the PFSeq idea with much less latency.

What's your take on the current state of the Android app market, regarding these kinds of real-time music composition tools? Are there particular tool sets or developers who I definitely need to check out? Who has the most solid "engine"?

Nice work on your AAudioTrack project. Haven't dug in, yet. Allocating my software juice on another project I need to get done. Looking forward to it, though.

well ones that come to mind are definitely FL Studio Mobile, AudioEvolution, and Mobile Studio Lite, and possibly Cubase if it exists for Android

but FL Studio team has a strong reputation as a high quality, professional DAW

FL Studio Team also has very detailed documentation available, examples include:

Audio Settings https://www.image-line.com/fl-studio-learning/fl-studio-online-manual/html/envsettings_audio.htm

Optimizing FL Studio performance https://www.image-line.com/fl-studio-learning/fl-studio-online-manual/html/app_opt.htm

CPU Meters & Multi-core CPU Processing https://www.image-line.com/fl-studio-learning/fl-studio-online-manual/html/toolbar_panels.htm#Multi-core_Processing

[mgood7123]

https://github.com/mgood7123/AAudioTrack/commit/02d96aaa4ddfa99027449e87d0d0dd451021f0bd this is the last commit in which the timing remains perfectly in sync

And https://github.com/mgood7123/AAudioTrack/commit/7fd3e7b4d70346fea45810380b5f5416d1728509 is when the timing still stays in sync, but can go temporarily out of sync

this appears to be caused by

        AAudioStreamBuilder_setBufferCapacityInFrames(builder, 16384);
        AAudioStreamBuilder_setFramesPerDataCallback(builder, 8192);

as removing that causes the beat to stay perfectly in sync

And with the large buffers it goes out of sync temporarily but catches up

[midenok]

Why can't you premix 400ms fragment from your sequence and play it instead of silence? Then continue playing sequencer from 400ms later.

[mgood7123]

"What's your take on the current state of the Android app market, regarding these kinds of real-time music composition tools? Are there particular tool sets or developers who I definitely need to check out? Who has the most solid "engine"?"

One would definitely be FL Studio Mobile, pretty solid audio engine

On Sun, 29 Nov 2020, 1:15 am peoplesfeelings, @.***> wrote:

Hm, very cool. Will be fun to dive back into the Android audio stuff, with C++, especially. I think you may be providing me with some good leads with which to redo the PFSeq idea with much less latency.

What's your take on the current state of the Android app market, regarding these kinds of real-time music composition tools? Are there particular tool sets or developers who I definitely need to check out? Who has the most solid "engine"?

Nice work on your AAudioTrack project. Haven't dug in, yet. Allocating my software juice on another project I need to get done. Looking forward to it, though.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/peoplesfeelings/PFSeq/issues/2#issuecomment-735243119, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGLITH6LP34MXAVNHWFAF5LSSEH2VANCNFSM4TZLOWNQ .

[mgood7123]

"Why can't you premix 400ms fragment from your sequence and play it instead of silence? Then continue playing sequencer from 400ms later."

Interesting

On Wed, 5 July 2023, 10:16 pm Aleksey Midenkov, @.***> wrote:

Why can't you premix 400ms fragment from your sequence and play it instead of silence? Then continue playing sequencer from 400ms later.

— Reply to this email directly, view it on GitHub https://github.com/peoplesfeelings/PFSeq/issues/2#issuecomment-1621640567, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGLITH4Z4GRICQOP257KXNDXOVLKLANCNFSM4TZLOWNQ . You are receiving this because you authored the thread.Message ID: @.***>