Make setLoopMode gapless on iOS

[ryanheise]

Note to self. On iOS, LoopingAudioSource is already gapless but setLoopMode should also be made gapless.

UPDATE: An implementation of this feature is now available for testing on the ios-gapless branch. The plan is to keep it on this branch until enough people have used it and reported that it is stable. If you run into any issues, please let me know below as that will help me to iron out any bugs before merging it and doing a public release.

[austinpower1258]

Could you also make setPlaybackRate gapless as well on iOS? Thanks!

[ryanheise]

@austinpower1258 unfortunately I don't think there is anything more I can do with setPlaybackRate. I do not pause and resume playback to do a playback rate change, I manipulate the playback rate on AVQuueuePlayer while it remains in the playing state.

Any quirks or distortions you hear when adjusting the playback rate I think may be a problem with iOS itself (unless you can see a coding mistake I have made?).

[RaphaelRevivor]

Hi, Any update on this? I am using _audioPlayer.setLoopMode(LoopMode.one) to play a single soundtrack in a loop. It would be nice to have this feature on iOS.

[ryanheise]

@austinpower1258 unfortunately I don't think there is anything more I can do with setPlaybackRate. I do not pause and resume playback to do a playback rate change, I manipulate the playback rate on AVQuueuePlayer while it remains in the playing state.

Any quirks or distortions you hear when adjusting the playback rate I think may be a problem with iOS itself (unless you can see a coding mistake I have made?).

This old message of mine doesn't make much sense :-) Perhaps I was accidentally replying to a different message on the wrong issue.

[ryanheise]

@RaphaelRevivor There is no update on this as there are some more urgent issues to fix first. However, there is a workaround for iOS. Using LoopingAudioSource is gapless, although it works by loading multiple copies of the same audio source, so you don't really want to use it for a large loop count. However, If you create a LoopingAudioSource with a loop count of 10, and then you also set the loop mode of the player to all, you'll get 10 gapless loops, then a small gap, then 10 more gapless loops, and so on.

[samry]

Thought I would post this in response to this thread, since it doesn't seem worthy of its own ticket (would be happy to fill it out if you think it is), but do you happen to know if there is a workaround to the small gap that occurs when using a LoopingAudioSource on many iOS Bluetooth devices? I have had multiple users complain that there is a small gap, or a short lag when using LoopingAudioSource on iOS with airpods or other bluetooth devices, but it doesn't occur with wired headphones. After testing it myself, I can confirm there's a short lag that occurs, but ONLY with bluetooth devices, regardless of the type of audio file (.wav, AAC, mp3, ac3, I've tried em all).

This isn't an issue on Android, and in the grand scheme of things isn't a huge deal, it's just that in my use-case I have back-to-back audio files playing that need to be seemless, so I've reverted back to fade-in/fade-outs and using several JustAudio instances that begin playback right before another one ends on iOS.

After doing some digging myself, I'm relatively certain it's just an iOS bug that you can't exactly get around, but curious if there are any workarounds you might recommend, assuming you've seen this issue before.

Again, would be happy to provide more info or fill out a ticket - thanks for your work on this project!

[ryanheise]

@samry I'm not aware of the Bluetooth issue, but you could try experimenting with the AVAudioSessionCategoryOptions in audio_session. Let me know if you have success with it.

[RaphaelRevivor]

@RaphaelRevivor There is no update on this as there are some more urgent issues to fix first. However, there is a workaround for iOS. Using LoopingAudioSource is gapless, although it works by loading multiple copies of the same audio source, so you don't really want to use it for a large loop count. However, If you create a LoopingAudioSource with a loop count of 10, and then you also set the loop mode of the player to all, you'll get 10 gapless loops, then a small gap, then 10 more gapless loops, and so on.

@ryanheise Thanks for the timely reply! But what you suggested here is probably not very suitable for our app. I will keep looking for other solutions

[samry]

@samry I'm not aware of the Bluetooth issue, but you could try experimenting with the AVAudioSessionCategoryOptions in audio_session. Let me know if you have success with it.

Couldn't get it to work, unfortunately, although some options (Allow Airplay/Bluetooth )seem to produce slightly less of a gap/lag than others. I'll look into it more if I have the time, but for now, will probably just loop the source files.

I figured there was a chance it just had to do with the resource-intensiveness of Flutter's debug mode, but users ran into it in production too. Thanks anyways!

[ryanheise]

I'd be interested to know if allowBluetoothA2dp does anything for you. I had intended to switch this on by default but it currently isn't.

[samry]

I'd be interested to know if allowBluetoothA2dp does anything for you. I had intended to switch this on by default but it currently isn't.

Still no cigar, unfortunately

[ryanheise]

OK, thanks for checking that. If you do discover any information, let me know and I'll implement it into the plugin.

[samry]

OK, thanks for checking that. If you do discover any information, let me know and I'll implement it into the plugin.

Just looked around a little bit this morning and I'm not sure if this is a viable solution, but this plugin seems to be using an AVPlayerLooper with an AVQueuePlayer as it's player (which just_audio uses, I think?).

I have absolutely no idea how that impacts the plugin and such, but thought it could be helpful - I'm pretty much lost when it comes to iOS, so I'll have someone on my team who is better at iOS look into it and see if they can put something together and will obviously share here if he can figure something out.

For now, we're just making loops in the actual source files, which isn't the worst thing in the world - better than a gap, at least, for our purposes.

[samry]

OK, thanks for checking that. If you do discover any information, let me know and I'll implement it into the plugin.

I can confirm that using the AVPlayerLooper works for getting no latency on Bluetooth looping playback.

[ryanheise]

Hi @samry , thanks for that information. It's interesting because AVPlayerLooper actually just uses AVQueuePlayer to do the work, so these should be the same, all else being equal. This makes me suspect that if AVPlayerLooper works, then not all else may be equal. When you say you can confirm AVPlayerLooper works, was it in the context of modifying just_audio or in a completely different codebase?

[samry]

@ryanheise so I've done a minimal reproduction with the just_audio plugin and the Ocarina (which uses AVQueuePlayer with AVPlayerLooper) plugin and the just_audio LoopingAudioSource has a small lag on iOS bluetooth (airpods, headphones, specifically - some speakers are okay) while the Ocarina plugin does not.

I'm going to have an iOS engineer take a look at it tomorrow - I don't have the objective C knowledge to really know what's going on, but fixing it would save a ton of space on our app. Will follow up if he finds anything

[RaphaelRevivor]

Any update on this?

[samry]

Agreed, would be huge if this was implemented. Still experiencing the small gap on iOS airplay/Bluetooth even after modifying audio_session and using LoopingAudioSource. Doesn't occur with OcarinaPlayer (I think it has to do with AvQueuePlayer, haven't dug into it for a while but lots of StackOverflow posts about the small gap). I'd be happy to file a bug report if helpful.

[ryanheise]

I will hopefully work on this issue next (as soon as the next releases of just_audio and audio_services are out). Apologies for the delay, it does really take a lot of time to implement each feature since there is a queue of them, and so pull requests are certainly welcome if people are motivated to see a particular feature implemented sooner than I can do it.

@samry the issue you describe is probably a separate issue since it affects only Bluetooth. I will try to fix the original issue first and we can see if that also corrects your issue.

[samry]

@ryanheise no worries at all - your plugin has been great so far. That small gap prevents us from using it 100%, but only in super small use cases on iOS only. I'll work on creating a replication project and documenting various forums where I've seen it come up.

I'm working on the development side of our app a lot less, but hopefully I can be of some assistance

[ryanheise]

@samry maybe #262 is related to your particular issue? If so, can you confirm you're seeing the same logs on that issue?

[samry]

@ryanheise yes, I can confirm I am seeing the same logs on LoopingAudioSource change with headphones:

onComplete advance to next: index = 1 On currentItem change, seeking back to zero ENTER BUFFERING: currentItem changed, seeking LEAVE BUFFERING: timeControlStatus LEAVE BUFFERING: currentItem changed, finished seek

I can also confirm that the latest commit works with regards to ConcatenatingAudioSource and LoopingAudioSource - it is gapless.

This is HUGE for us - thank you so much!!!

[ryanheise]

Great! That will not be the final version I'll release, though. I'd like to set an appropriate threshold for the delta that works for Bluetooth. Since #262 perfectly matches your issue, you can follow that issue for further developments, or contribute your own deltas to the discussion, and I'll use this issue over here just for the loop mode issue.

[ryanheise]

To all those who want gapless looping on iOS, I have created an experimental branch called ios-gapless.

Please have a try and let me know if you notice any issues.

One known issue is that it probably will run into problems if you loop a single very short item because it won't have enough time to buffer the next loop item.

[RaphaelRevivor]

I can confirm that the code in branch ios-gapless works for me. I tested with several sound tracks with length ranging from 20s to 35s and they all work. And BTW I used local assets (setAsset() method). Great job! Just out of curiosity, how short the item needs to be to make it stop working?

[ryanheise]

@RaphaelRevivor thanks for testing! I plan to keep this as a separate branch for a while. If enough people use it, test it and report it as being stable, I'll merge it into master.

Just out of curiosity, how short the item needs to be to make it stop working?

This is purely based on theory and I don't have any hard numbers. But in principle, particularly if the audio is from a remote source, there will not be enough time between each loop for the next item to be prepared and probably you'll just experience some stalling.

[ryanheise]

Note: I've just merged in the latest code from master into ios-gapless.

[RaphaelRevivor]

@ryanheise I see, thanks! Hope there will be more people testing this. (From my personal point of view, merging would be the earlier the better :) )

[samry]

Works flawlessly for me.

The smallest audio clips I'm loading are about 5 seconds in length, from assets only.

No gap on Bluetooth, airplay, or speakers.

[fuzing]

IOS Gapless looping seems to be working well -

15-30 second audio clips from local storage (IOS 12/13) - no problems detected.

15-30 second audio clips from network (IOS 12/13) - working well. In older versions of just_audio, looping a network source would hit the network for each iteration (i.e. there did not appear to be any caching). With this branch, I've experimented with disconnecting the IOS device from the network (turning wifi on the device off) and a single audio loop keeps playing. Is this true for any loop (e.g. a 60 minute loop?)? - It would be great to get a brief explanation of the caching strategy in play (caching entire playlist? caching just first/last track? any other info so as to avoid memory exhaustion issues would be very useful/helpful).

Well done, and thank you for this - this is a really welcome update - Happy Holidays!

PB

[ryanheise]

Thanks everyone for testing! Let's see how it goes after some more use.

@fuzing The iOS implementation still has room for improvement, but frankly the AVQueuePlayer is a bit of a black box. I am not 100% sure of when it starts filling or emptying its buffers.

If AVQueuePlayer doesn't do this automatically (like it does in ExoPlayer), I will need to eventually add some more complicated programming to force this to happen myself, and this behaviour could be tied to the ConcatenatingAudioSource.useLazyPreparation. At the very least, the documentation does recommend against what I am doing. Ideally, an app should not add all playlist items to the AVQueuePlayer, it should manually implement a sliding window.

Looping has been implemented as a sort of treadmill. If the normal queue is A-B-C-D, what LoopMode.all does is enqueue A-B-C-D-A1 where A1 is a clone of A with its own buffers. With LoopMode.one, it enqueues A-A1. As soon as A1 is reached, I flip A-A1 around so that A1 becomes A, and I dynamically adjust the queue leaving the new A in place while adding items after it: A-B-C-D. Then what happens to the A1 that was just flipped and removed, it gets added to the end of the treadmill resulting in A-B-C-D-A1 (or A-A1 depending on the loop mode).

In order to achieve gapless playback when looping back to the same item, it is necessary to load the item twice with two separate buffers, buffering two different positions of the media. One improvement with extremely short media is to actually have multiple duplicates lined up. E.g. Looping a sub-second item would probably need something like this: A-A1-A2-A3-A4-A5 with 5 duplicates all stacked up and ready to go.

Now, as a black box, I am not sure why AVQueuePlayer would require a network connection in the old implementation and not in the new implementation. In both implementations, I have to seek back to the beginning after an item finishes playing.

But it is definitely worth investigating what happens to a buffer after playback of an item has completed. In the LoopMode.one scenario, as soon as an item is completed, it is "immediately" added to the end of the treadmill and maybe the AVQueuePlayer black box does something there to keep it in memory. In the normal non-looping scenario, as soon as playback of an item is finished, it is removed from the player. It needs to be investigated whether its internal buffers are emptied at that point, or whether I can do something to force the buffer to be emptied.

[fuzing]

Hi Ryan -

Thanks for your thoughtful response - that makes perfect sense. No doubt the AVQueuePlayer has its own caching/eviction strategy - I'll see if I can find any docs on this - it's possible that the native player may have used a different strategy based on current memory consumption or some other variable - we've reduced our app memory footprint which may provide more headroom for caching (who knows? As you say - much of this is "Black Box"). FYI, as indicated I did do testing on both local storage and network audio files, and didn't see any obvious issues - including things like memory leakage. I also tested down to sub 2s audio durations and did not observe any stalling/stuttering (although I'm testing on a very low-latency high-bandwidth connection, not over cellular).

From my perspective this is a big win, so thank you again for all of your hard work on this and your other projects - we very much appreciate this,

Take care,

PB

[ryanheise]

No doubt the AVQueuePlayer has its own caching/eviction strategy - I'll see if I can find any docs on this

That would certainly be helpful. Thanks!

[ryanheise]

I have just merged in the latest 0.6.4 code into the ios-gapless branch. Please try out this version if you can, and if it remains stable until the weekend, I will aim to release this on the weekend.

[RaphaelRevivor]

Hi Ryan,

I just tested with the most recent changes in branch ios-gapless. It works fine for me. Again good job!

HNY and Stay Safe, Raphael

[jjb182]

Hi Ryan (et al) I was wondering if you ever thought of setting up a just_audio FB group for simple questions like what I'm about to ask.

I am currently using the main branch of 0.6.4, and on iOS _player.setLoopMode(LoopMode.one) does not work when I turn off the screen. Playlists work great when loopMode IS off, but not set to .one.

Is this an iOS limitation, or do I have something wrong with my code? I'm about to switch to the new branch and see if that changes anything.

Next Also, I want to play (loop) a local file but I don't see a direct replacement for _player.setFilePath(path) to be used with ConcatenatingAudioSource. _player.setFilePath works great when I'm Not using ConcatenatingAudioSource.

**Using ConcatenatingAudioSource I attempted to create a playlist with AudioSource.uri(Uri.parse(Local_Path)), but this fails. Attempting to set the local file like this also fails await _player.setAudioSource(ProgressiveAudioSource(Uri.parse(Local_Path))); both with [flutter: An error occured (-1002) unsupported URL]

Thanks

var saveDir = _settings.savePath; var path = saveDir + Platform.pathSeparator + 'frequency_1.wav'; var path2 = saveDir + Platform.pathSeparator + 'frequency_2.wav'; var path3 = saveDir + Platform.pathSeparator + 'frequency_3.wav'; var path4 = saveDir + Platform.pathSeparator + 'frequency_4.wav';

ConcatenatingAudioSource _playlist =
    ConcatenatingAudioSource(children: [
  AudioSource.uri(Uri.parse(path)),
  AudioSource.uri(Uri.parse(path2)),
  AudioSource.uri(Uri.parse(path3)),
  AudioSource.uri(Uri.parse(path4)),
]);

try {
  await _player.setAudioSource(_playlist);
} catch (e) {
  // catch load errors: 404, invalid url ...
  print("An error occured $e");
}

_player.setLoopMode(LoopMode.one);

try {
  _player.play();
} catch (e) {
  // catch load errors: 404, invalid url ...
  print("An error occured $e");
}

════════ Exception caught by services library ══════════════════════════════════════════════════════ MissingPluginException(No implementation found for method cancel on channel com.ryanheise.just_audio.events.7c66ef98-9a74-44c5-acae-882a913e8181) ════════════════════════════════════════════════════════════════════════════════════════════════════ flutter: An error occured (-1002) unsupported URL

[andrei-pavel]

setting up a just_audio FB group

Or a Discussions section on Github

[jjb182]

Hi Andrei, or Ryan, or anybody who is up right now. Does player.setLoopMode(LoopMode.one) work on iOS when the screen is off? In the example code that I copied for 0.4.4 the playlist plays fine in the background, but when I loop a track (using LoopMode.one) and turn off the screen (these are short files--5 sec) it plays that one file and then quits. Same with 0.6.4.... I thought that maybe if I pulled the new gapless branch that it might work, but I'm not Flutter savvy enough to make that work yet.

Anyhow, LoopingAudioSource(Count: xx, ... works great. But LoopMode.one would be more convenient.

LoopingAudioSource( count: 10,

I'm pulling my hair out. Thanks

On Fri, Jan 1, 2021 at 10:36 PM Andrei Pavel notifications@github.com wrote:

setting up a just_audio FB group

Or a Discussions section on Github

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ryanheise/just_audio/issues/137#issuecomment-753425525, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU3WN7MRRU5QIEIHWJ4WZLSX2ICPANCNFSM4PXHQGVA .

[ryanheise]

(Note this is a bit off topic but just to quickly address it here, and then provide a recommendation on where this separate discussion could continue) GitHub discussions has been proposed before on audio_service, and I'll copy and paste my response here FYI since I would have the same exact thoughts for just_audio:

Hmm... It's interesting to think about. Here are some of my initial thoughts about its different uses:

• Q&A: I think StackOverflow is probably better. It has a good system for filtering question quality and eliminating duplicate questions. Being a separate website may also be a healthy way to prevent a single company from handling everything ;-)
• Ideas: I like ideas discussions, although our current system of creating a Feature Request issue seems to suffice, and in some ways is better: It's all under the same issues page so I can easily manage what issues I'm currently working on through the label filtering system.
• General: It's true I don't have a spot for this yet but I would prefer to have specific categories anyway.
• Show & Tell: I like this one. It would be great to hear what people are doing with audio_service.

It may be that bringing Q&A over to here from StackOverflow will help to build critical mass and thereby make it easier for the Show & Tell thread to gain traction (as opposed to if Show & Tell were the only discussion category there). It also may lower the resistance of people to follow the link to an external website for asking questions, and in effect, could lower the burden to close invalid issues. If I use discussions, it would make sense to use the Q&A and Show & Tell categories.

The only downside then is that it would be bad for StackOverflow if everybody follows suit and moves their Q&A away from a well-established site and quality site. I might wait a bit to see how others may (or may not) choose to adopt it.

My current inclination is to continue using StackOverflow. It is where I currently answer questions of this sort, and it does a wonderful job of encouraging people to ask questions in a way that actually helps answerers to answer them. I do check StackOverflow regularly although I don't guarantee that I answer every question.

In any case, this is a bit off topic for the current issue which I would prefer to keep focused on the testing and development of the ios-gapless branch. If you would like to propose GitHub Discussions, you can create a "New issue" and maybe "Feature request" would be the most appropriate classification since it is an idea for how to improve the project.

But also please do consider using StackOverflow as it is what is currently linked to from the "New issue" page, and it is my favoured system.

[ryanheise]

This has finally been released in 0.6.5. Thanks everyone for your patience, and most of all for helping to test this re-implementation of looping.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs, or use StackOverflow if you need help with just_audio.