[iOS] Car Play does not allow to start playback (no media items loaded)

[glettl]

Which API doesn't behave as documented, and how does it misbehave? iOS Car Play (one-isolate)

Minimal reproduction project Provide a link here using one of two options: The example Copied the example code in my app, as Car Play entitlements and a provisioning profile is needed.

To Reproduce (i.e. user steps, not code) Steps to reproduce the behavior:

1. Build and Launch app
2. Click on the Car Play App Icon
3. App crashes

Error messages Simulator System Log:

CarPlayTemplateUIHost[63568]: assertion failed: 20G80 18E182: libxpc.dylib + 50305 [454053CA-B690-3B99-BE31-2369B04EA493]: 0x7d
Runner[63571]: assertion failed: 20G80 18E182: libxpc.dylib + 50305 [454053CA-B690-3B99-BE31-2369B04EA493]: 0x7d
Runner[63571]: assertion failed: 20G80 18E182: libxpc.dylib + 35928 [454053CA-B690-3B99-BE31-2369B04EA493]: 0x87

Expected behavior Car Play Screen opens and shows Media Items returned by the getChildren method of the AudioHandler. Like it works on Android Auto: app shows playable titles/streams on the Car Play display and user is able to start playback.

Screenshots no relevant info on screen for iOS 14.5

iOS 13.3:

only works when playback is started on iPhone

Runtime Environment (please complete the following information if relevant):

• Device: iPhone 12 Simulator
• OS: 14.5
• Device: iPhone 11 Simulator
• OS: 13.3

Flutter SDK version

Flutter (Channel unknown, 2.2.1, on macOS 11.5.1 20G80 darwin-x64, locale en-AT)

Additional context

[glettl]

I did further research and it seems, that the com.apple.developer.carplay-audio (Car Play Framework, introduced with iOS 14.x) does not work, as it needs custom screens. However, if the com.apple.developer.playable-content entitlement is still possible to use which uses predefined screens (as on Anrdoid Auto). The com.apple.developer.playable-content is completely fine for my use case, if it would be able to start the playback on the Car Play Screen.

[jmshrv]

Just wondering, what did you have to do to get CarPlay working? I'm confused on what getChildren and subscribeToChildren means

[ryanheise]

The documentation at the moment is a bit scarce but these methods are specifically for Android Auto. CarPlay support is currently only implicit in that iOS has been reported to implicitly broadcast some state to CarPlay.

[glettl]

Just wondering, what did you have to do to get CarPlay working? I'm confused on what getChildren and subscribeToChildren means

First, I had to request the missing entitlements from Apple and to use a custom signing profile.

Second, I implemented MPPlayableContentManager for my needs and I used a custom MethodChannel that calls required functions of the AudioHandler (in my case getChildren) to avoid duplicate code. I've just extended the Flutter AppDelegate with using a swift extension. I consulted this tutorial.

After these two steps the app was visible on the CarPlay screen and it was possible to play audio.

Here's my code, however it is very specific and tailored to my needs.

import Foundation
import MediaPlayer

extension AppDelegate {

    func setupCarPlay() {
        playableContentManager = MPPlayableContentManager.shared()

        playableContentManager?.delegate = self
        playableContentManager?.dataSource = self

        playableContentManager?.beginUpdates();
        self.streamHolder.load(methodChannel: methodChannel!, completion: { error in
            self.playableContentManager?.endUpdates();
        })

    }
}

extension AppDelegate: MPPlayableContentDelegate {

    func playableContentManager (_ contentManager: MPPlayableContentManager, initiatePlaybackOfContentItemAt indexPath: IndexPath, completionHandler: @escaping (Error?) -> Void) {
        if let stream = self.streamHolder.streams?[indexPath[1]] {
            methodChannel?.invokeMethod("playStream", arguments: stream.value(forKey: "id"))
        }

        completionHandler(nil)
    }

    func beginLoadingChildItems(at indexPath: IndexPath, completionHandler: @escaping (Error?) -> Void) {
        print("beginLoadingChildItems")
        self.streamHolder.load(methodChannel: methodChannel!, completion: { error in
            completionHandler(error)
        })
    }
}

extension AppDelegate: MPPlayableContentDataSource {

    func numberOfChildItems (at indexPath: IndexPath) -> Int {
        print("numberOfChildItems \(indexPath.indices.count)")
        if indexPath.indices.count == 0 {
            if(self.streamHolder.streams == nil) {
                self.streamHolder.load(methodChannel: methodChannel!, completion: { error in
                })
            }
            return 1
        }
        if(self.streamHolder.streams == nil) {
            self.streamHolder.load(methodChannel: methodChannel!, completion: { error in
                self.playableContentManager?.reloadData();
            })
        } else {
            return self.streamHolder.streams?.count ?? 0;
        }
        return 0;
    }

    func contentItem(at indexPath: IndexPath) -> MPContentItem? {
        if indexPath.count == 1 {
            // Tab section
            let item = MPContentItem(identifier: "Player")
            item.title = "Player"
            item.isContainer = true
            item.isPlayable = false
            print("image")
            if let tabImage = UIImage(named: "CarPlayTabIcon") {
                print(tabImage);
                item.artwork = MPMediaItemArtwork(boundsSize: tabImage.size, requestHandler: { _ -> UIImage in
                    return tabImage
                })
            }
            return item
        } else if indexPath.count == 2, indexPath.item < self.streamHolder.streams?.count ?? -1 {
            let stream = self.streamHolder.streams?[indexPath.item]
            let item = MPContentItem(identifier: stream?.value(forKey: "id") as! String)
            item.title = stream?.value(forKey: "album") as! String
            item.subtitle = "My Stream"
            item.isPlayable = true
            item.isStreamingContent = true
            if let artUri = stream?.value(forKey: "artUri") {
                print(artUri)
                ImageLoader.sharedLoader.imageForUrl(urlString: artUri as! String) { image, _ in
                    DispatchQueue.main.async {
                        guard let image = image else { return }
                        item.artwork = MPMediaItemArtwork(boundsSize: image.size, requestHandler: { _ -> UIImage in
                            return image
                        })
                    }
                }
            }
            return item
        }
        return nil

    }
}

[ryanheise]

Nice, @glettl ! If this is generally useful (which I think it would be), then let's try to incorporate it in audio_service. Would you be interested in making a pull request? Even if not, I will leave this issue open with your code snippet since it would eventually help guide an implementation of this in the plugin.

[glettl]

I completely agree. However, as mentioned above, this is a solution tailored to my needs. In other words a lot of the functionality from the flutter/dart part is missing in my implementation. There is no possibility to create a folder structure (because I do not need it right now). I even do not know all features of MediaItem (which are the items displayed on the CarPlay Screen).

Long story short, I could try to implement this solution, but it will not satisfy all users - it will definitely cause a lot of bug issues.

To implement this profoundly, there has to be an example project as basis which covers all the functionality.

[ryanheise]

To implement this profoundly, there has to be an example project as basis which covers all the functionality.

Do you mean like example/lib/example_multiple_handlers.dart? This example demonstrates the corresponding feature on Android, by overriding the getChildren and subscribeToChildren methods. It seems from my understanding of your code above that these are similar and can be mapped between each other, or if there are any irreconcilable differences, I would be happy to change the underlying code in audio_service to either make them compatible, or even to create two different APIs, one to support Android Auto-specific media browsing and another to support CarPlay-style media browsing.

[vanlooverenkoen]

@glettl do you have a small example, we are looking for something similar.

We also have been looking to use: https://pub.dev/packages/flutter_carplay but that results in a completely new implementation.

[vanlooverenkoen]

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_playable-content?language=objc

Is deprecated. (So it can only be used from iOS 12 to iOS 14)

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_carplay-audio?language=objc

Should be used after that.

[vanlooverenkoen]

I can't get this to work, at all. Even when switching to iOS 12.4 it does not work and I constantly get

Unable to connect to "AppName"

There is a problem loading this content

And pages that need 20-30 seconds of loading.

this is my implementation

import Foundation
import MediaPlayer

extension AppDelegate {

    func setupCarPlay() {
        playableContentManager = MPPlayableContentManager.shared()
        playableContentManager?.delegate = self
        playableContentManager?.dataSource = self
        playableContentManager?.beginUpdates()
    }
}

extension AppDelegate: MPPlayableContentDelegate {

    func playableContentManager(_ contentManager: MPPlayableContentManager, initiatePlaybackOfContentItemAt indexPath: IndexPath, completionHandler: @escaping (Error?) -> Void) {
       print("playableContentManager")
       DispatchQueue.main.async {
           completionHandler(nil)
           #if targetEnvironment(simulator)
               UIApplication.shared.endReceivingRemoteControlEvents()
               UIApplication.shared.beginReceivingRemoteControlEvents()
           #endif
       }
    }
}

extension AppDelegate: MPPlayableContentDataSource {

    func beginLoadingChildItems(at indexPath: IndexPath, completionHandler: @escaping (Error?) -> Void) {
        print("beginLoadingChildItems")
        completionHandler(nil)
    }

    func numberOfChildItems(at indexPath: IndexPath) -> Int {
        print("numberOfChildItems")
        if indexPath.indices.count == 0 {
            return 3
        }
        return 10
    }

    func contentItem(at indexPath: IndexPath) -> MPContentItem? {
        print("contentItem")
        if indexPath.count == 1 {
            let section = indexPath[0]
            let item = MPContentItem(identifier: "tab-\(indexPath.section)")
            item.title = "Tab-\(indexPath.section)"
            item.isContainer = true
            item.isPlayable = false
            return item
        }
        if indexPath.count == 2 {
            let item = MPContentItem(identifier: "page-\(indexPath.section)")
            item.title = "Boek-\(indexPath.section)-\(indexPath.row)"
            item.subtitle = "Subtitle"
            item.isPlayable = false
            item.isContainer = true
            return item
        }
        if indexPath.count == 3 {
            let item = MPContentItem(identifier: "detail-\(indexPath.section)-\(indexPath.row)-\(indexPath.item)")
            item.title = "Detail-\(indexPath.section)-\(indexPath.row)-\(indexPath.item)"
            item.subtitle = "Subtitle"
            item.isPlayable = true
            return item
        }
        return nil
    }
}

@glettl can you give me some guidance. I will now check the new iOS 14 implementation.