[HELP WANTED] iOS control center: testers and fixers

[ryanheise]

Help Wanted

We are preparing the upcoming 0.18 release of audio_service on the one-isolate branch with a number of performance, feature and API improvements.

The last critical piece of the puzzle is to restore the iOS/macOS control center functionality.

If you would like to help, the more eyes on it the faster we should be able to make progress. As a tester, you could help by checking out via git older versions of the code on the one-isolate branch and running them to see if the control center was working, and in this way identify which specific commit/s broke the control center. That in turn will help us identify the relevant code and come up with a fix. If you find a commit where something broke, please share the commit reference below.

If you have some experience with iOS programming, you can help in other ways by reviewing and debugging the code. Some ideas of what to look into can be found at the bottom of this bug report.

Which API doesn't behave as documented, and how does it misbehave?

playbackState and mediaItem state changes are forwarded to the iOS/macOS platform side but they do not consistently result in changes to the control center and now playing info.

Minimal reproduction project

The example on the one-isolate branch exhibits the issue as is.

To Reproduce (i.e. user steps, not code)

Steps to reproduce the behavior:

1. Click play
2. Open the control center to check the displayed state
3. Click other controls in the Flutter app
4. Check if they are reflected in the control center

You may find one of various problems. E.g. the first state update is reflected in the control center but subsequent ones aren't. Or you may find that the buttons are disabled. Or you may even find that the first state update isn't reflected.

Error messages

None

Expected behavior

Actions in the Flutter UI that trigger state changes should be reflected appropriately in the control center.

Screenshots

None

Runtime Environment (please complete the following information if relevant):

• Device: Found on macOS Big Sur, and iPhone 12 Pro Max "Simulator" (iOS 14.4)
• OS: Running on macOS Big Sur

Flutter SDK version

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 2.0.5, on macOS 11.2.3 20D91 darwin-x64, locale en-AU)
[✗] Android toolchain - develop for Android devices
    ✗ ANDROID_HOME = /Users/ryan/opt/android-sdk
      but Android SDK not found at this location.
[✓] Xcode - develop for iOS and macOS
[✗] Chrome - develop for the web (Cannot find Chrome executable at /Applications/Google Chrome.app/Contents/MacOS/Google
    Chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[!] Android Studio (not installed)
[✓] Connected device (2 available)

Additional context

Here are some potential things to look into:

• nowPlayingInfo race condition can prevent updates: https://stackoverflow.com/questions/34867294/mpnowplayinginfocenter-nowplayinginfo-not-updating-at-end-of-track
• In the past, flutter_tts has prevented the control center from working - we could test removing that
• We can look at the interaction of MPRemoteCommandCenter, MPNowPlayingInfoCenter and AVAudioSession to ensure they're all working in coordination. E.g. checking that the timing of these calls works together.
• We can look into specific MPRemoteCommands that need to be enabled for things to work correctly.
• We can log calls to the MPRemoteCommandCenter, MPNowPlayingInfoCenter and AVAudioSession APIs in both versions of the code and try to spot any differences that could explain the behaviour.
• If fortunate enough, we can use experience/knowledge to point out directly what is wrong.

[kmod-midori]

The system can certainly handle updates every several seconds, although I have no clue if one second works.

Many Chinese music apps (ab)used the cover artwork to show lyrics on the lock screen by setting a new cover with lyrics rendered on it every time the lyric changes (source: https://www.zhihu.com/question/33634376/answer/124495068). So either Apple introduced new bugs later on or we have more problems on our side?

[nt4f04uNd]

It seems that it happens only on macOS. I'd prefer not to make unnecessary calls to update notification anyways. Though, it's likely that the effect of updating it each second is probably neglectable (if we take out this macOS bug with artwork)

[esiqveland]

I think it would be preferable to only update nowPlayingInfo when needed, but it might require a separate call to support seekTo from Flutter side (jumps in playback position), or a tighter specification on how important it is to use updateTime and updatePosition correctly.

[esiqveland]

Reading further, some also suggest to keep a local instance of nowPlayingInfo and mutate that. Perhaps only allocating a new instance when changing mediaitem. If the plugin is single-threaded, we could re-use the instance and only update the fields that actually changed from last update.

[ryanheise]

@nt4f04uNd I've left some comments on #686, and we can continue the discussion over there.

[ryanheise]

or a tighter specification on how important it is to use updateTime and updatePosition correctly.

This is something that could also be improved in just_audio, so that it updates updateTime only on actual time discontinuities. It's currently a bit eager on the Android side, and a bit more sane on the iOS side

[ryanheise]

I've merged #686 which should address the main issues with the control center not updating. On macOS, it uses the playbackState property although I'm surprised it doesn't require an @available check. It's also confusing that the Swift and Objective C documentation for this API say that it is available in different versions of iOS. Now that is merged, I would be interested for people to test this out on iOS with different deployment targets and see if you get any compilation errors about this API not being supported.

We still need to take a look at the updateTime calculation, we can still optimise further the update of nowPlayingInfo so that it happens only when something has changed, and we may have a "possible" regression where the album art disappears when clicking pause (I say "possible" because I thought I fixed it earlier by eliminating some unnecessary duplicate updates - although I could be wrong). - edit: That last point was observed on macOS, not iOS.

The last two points may be related in that the artwork may mess up if we update the nowPlayingInfo too frequently. On this line:

center.nowPlayingInfo = nowPlayingInfo;

We should probably wrap it in an if that checks if any of the properties were actually updated before setting the new object.

Reading further, some also suggest to keep a local instance of nowPlayingInfo and mutate that. Perhaps only allocating a new instance when changing mediaitem. If the plugin is single-threaded, we could re-use the instance and only update the fields that actually changed from last update.

That is something I experimented with although I didn't notice any behavioural changes. I'll give it another try now after the above merge.

[ryanheise]

we can still optimise further the update of nowPlayingInfo so that it happens only when something has changed, and we may have a "possible" regression where the album art disappears when clicking pause (I say "possible" because I thought I fixed it earlier by eliminating some unnecessary duplicate updates - although I could be wrong). - edit: That last point was observed on macOS, not iOS.

The last two points may be related in that the artwork may mess up if we update the nowPlayingInfo too frequently. On this line:

Just checked with some logging, and updateNowPlayingInfo is called twice in succession when the album art disappears, so it does look like the same issue that I worked on before. I'm adding some code to optimise away unnecessary updates, but I also noticed that the playback position is also updated anyway forcing the change (which can probably be traced back to just_audio).

So To completely fix this, I think we'll also need some just_audio optimisations, but more importantly we need a throttling mechanism in audio_service to prevent updates to nowPlayingInfo faster than one second.

[ryanheise]

Pushed 1 commit which optimises away unncessary nowPlayingInfo updates. Will look at the throttling and just_audio issues some time later.

[esiqveland]

I have tested #686 well on iOS, seems to work much better with no drawbacks. Tested briefly on macOS with #686 and with https://github.com/ryanheise/audio_service/commit/9f9747461a9ecd043ab3e122c98b78752f6abba4

It works much better with macOS control center, the only thing I found is the artwork disappears when you press pause, like before. Looking at the logs, I think there actually might be some intermediate state that breaks the cover art.

[shripal17]

Hi,

With the latest HEAD, I can confirm that: a. seeking now works properly, both from/to app/notif centre b. Play/Pause button is no longer disabled and works as expected.

However, seek forward/previous buttons are still broken as per my last comment here.

Also, first song/music's details of the playlist are not updated in the notif centre. It shows default iOS media notification without specific details of the song/music

[ryanheise]

the only thing I found is the artwork disappears when you press pause, like before. Looking at the logs, I think there actually might be some intermediate state that breaks the cover art.

Last time I fixed it by avoiding to call updateNowPlayingInfo too quickly in succession, hence why I'd like to add the throttling. But if you have another theory I'd be interested also to find out what intermediate state you think may be breaking it.

However, seek forward/previous buttons are still broken as per my last comment here.

Thanks for the reminder, I haven't had a chance to test this yet.

Also, first song/music's details of the playlist are not updated in the notif centre. It shows default iOS media notification without specific details of the song/music

Are you able to make a video of this behaviour?

[shripal17]

Are you able to make a video of this behaviour?

Yes, I will post a screen-record tomorrow, as a temporary fix, I have manually added the song to mediaItem on queue load and to fix the current position issue, I also seek AudioPlayer manually to Duration.zero on first song play

[shripal17]

Hi,

Just discovered that seeking in iOS is done by long pressing the seek buttons from notif centre! I had no idea cause I don't have an iPhone and I haven't used iOS much.

So, the seeking works as it should. But, nothing happens on single tap of these buttons. Ideally, it should call seekToNext and seekToPrevious

PFA the screen-recording taken from iPhone SE 2nd Gen iOS14.3 Simulator

https://mega.nz/file/Wd5jTCiC#Sl7kPfn8Ho4BBOelkZW-tMCXmsOJ3ebTzC1Rif9cnv8

[nt4f04uNd]

Could you please attach a minimal reproducible sample, or try if it works in the example app for you?

[shripal17]

Let me try the example app first

[shripal17]

Also, another observation/bug:

Song with album art is played -> Skipped to next song which doesn't have album art (artUri is passed as null) -> Notif centre still has old app's album art, ideally it should be empty.

Actual song with no album art: Notif centre still shows album art of the previous song:

[shripal17]

Could you please attach a minimal reproducible sample, or try if it works in the example app for you?

It's working with the example app! In my app, I haven't overriden the skipToNext and skipToPrevious methods because initially I followed master branch's integration instructions and it said skipToQueueItem would automatically be called if either of the methods are not overriden.

[nt4f04uNd]

It should, if you mixed-in QueueHandler

[shripal17]

Oh, right! I forgot to do that, sorry for the trouble.

So now, it narrows down to just the album art issue, otherwise ios implementation is quite stable now.

[ryanheise]

Regarding the album art issue(s), I'll be working on the throttling (see discussion above) over the weekend which should hopefully help with this.

[ryanheise]

Just did a quick sanity check using throttleTime() on each stream and it did not solve the disappearing artwork problem on macOS.

While playing, the artwork is visible. When clicking pause, the artwork disappears, and the following is logged after the pause:

2021-05-09 01:16:17.249 example[2226:34450] XXX: broadcasting state
2021-05-09 01:16:17.249 example[2226:34450] ## updateControl 1 enable=0
2021-05-09 01:16:17.249 example[2226:34450] ## updateControl 2 enable=1
2021-05-09 01:16:17.249 example[2226:34450] ### MPNowPlayingInfoPropertyPlaybackRate = '0'
2021-05-09 01:16:17.249 example[2226:34450] ### MPNowPlayingInfoPropertyElapsedPlaybackTime = '947.6130000000001'
2021-05-09 01:16:17.249 example[2226:34450] ### updating nowPlayingInfo

I've tried eliminating some of these updates:

2021-05-09 01:16:17.249 example[2226:34450] XXX: broadcasting state
2021-05-09 01:16:17.249 example[2226:34450] ### MPNowPlayingInfoPropertyPlaybackRate = '0'
2021-05-09 01:16:17.249 example[2226:34450] ### updating nowPlayingInfo

Still, the artwork disappears.

Finally, I tried eliminating the change to the playback rate itself, and bingo! The artwork doesn't disappear. I guess that sort of makes sense, since on macOS it relies on MPNowPlayingInfoCenter.playbackState instead. Will hopefully do some more experimentation and a fix tomorrow.

[ryanheise]

Finally, I tried eliminating the change to the playback rate itself, and bingo! The artwork doesn't disappear. I guess that sort of makes sense, since on macOS it relies on MPNowPlayingInfoCenter.playbackState instead. Will hopefully do some more experimentation and a fix tomorrow.

After more experimentation this does not work reliably (unfortunately). I guess there are still some mysteries on the macOS side...

Song with album art is played -> Skipped to next song which doesn't have album art (artUri is passed as null) -> Notif centre still has old app's album art, ideally it should be empty.

Let's see if I can have more success with this issue.

[ryanheise]

The latest commit should now respect media items that have no artwork.

[shripal17]

The latest commit should now respect media items that have no artwork.

Great news! I'll test this tomorrow and let you know if it fixes the issue

[girish54321]

This is amazing.😀😃 @ryanheise (iPhone 11 "14.3")

[nt4f04uNd]

We should follow https://developer.apple.com/documentation/mediaplayer/becoming_a_now_playable_app/ to check we didn't miss anything

[xysun]

one thing I noticed for the tts example is, the control center is disabled (in grey) whenever code is in sleeper.sleep part. You can see it more obviously if you change the sleep to slightly longer, eg. to 5 seconds, and enter lock screen.

(I noticed this on main branch first so was testing to see if the same problem happens on this one-isolate branch)

Is there a way we can keep the control active during sleep? They are active in the in app controls.

[ryanheise]

This is a known limitation of iOS and probably something I could mention in the readme. The way around it is to play a silent audio file for the duration you want to keep the phone awake. This will register in iOS as a legitimate background activity.

[ryanheise]

Thanks to everyone who helped both with testing and coding. 0.18.0 is now released!

I'll close this issue since it's more of an umbrella issue, but any remaining issues with the iOS/macOS control center can be opened as new issues.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs, or use StackOverflow if you need help with audio_service.