How to switch the adaptation set (trackGroup) associated with a video renderer on the fly without restarting the player

[Shrihari1428]

Hi Everyone,

I am creating an implementation of exoplayer where I have 4 video renderers (created 3 extra). I have modified my DASH manifest in such a way that each adaption set represents a camera angle and one adaptation set for audio. So now I assign each video adaptation set (video trackGroup) to a video renderer. So if there are 6 video adaptation sets, 4 renderers are initialised with 4 adaptation sets initially. Now I want to switch the adaption set associated with a renderer with an adaptation set thats not in use by specifying the trackGroup id and the renderer index . How do I do this?

Below is the implementation of my CustomTrackSelector

public class CustomTrackSelector extends DefaultTrackSelector {

    public CustomTrackSelector(Context context) {
        super(context);
    }

    @NonNull
    @Override
    protected ExoTrackSelection.@NullableType Definition[] selectAllTracks(
            @NonNull MappedTrackInfo mappedTrackInfo,
            @NonNull @RendererCapabilities.Capabilities int[][][] rendererFormatSupports,
            @NonNull @RendererCapabilities.AdaptiveSupport int[] rendererMixedMimeTypeAdaptationSupports,
            @NonNull Parameters params
    ) throws ExoPlaybackException {
        int rendererCount = mappedTrackInfo.getRendererCount();
        ExoTrackSelection.@NullableType Definition[] definitions =
                new ExoTrackSelection.Definition[rendererCount];
        // Custom change start
        // Get multiple selected videos if renderers available
        @Nullable
        ArrayList<Pair<ExoTrackSelection.Definition, Integer>> selectedVideos =
                selectVideoTracks(
                        mappedTrackInfo,
                        rendererFormatSupports,
                        params,
                        (int rendererIndex, TrackGroup group, @RendererCapabilities.Capabilities int[] support) ->
                                VideoTrackInfo.createForTrackGroup(
                                        rendererIndex, group, params, support, rendererMixedMimeTypeAdaptationSupports[rendererIndex]),
                        VideoTrackInfo::compareSelections
                );
        if (selectedVideos != null) {
            for (Pair<ExoTrackSelection.Definition, Integer> selectedVideo: selectedVideos) {
                @Nullable
                Pair<ExoTrackSelection.Definition, Integer> selectedImage =
                        params.isPrioritizeImageOverVideoEnabled || selectedVideo == null
                                ? selectImageTrack(mappedTrackInfo, rendererFormatSupports, params)
                                : null;

                if (selectedImage != null) {
                    definitions[selectedImage.second] = selectedImage.first;
                } else if (selectedVideo != null) {
                    definitions[selectedVideo.second] = selectedVideo.first;
                }
            }
        }
        // Custom change end
        @Nullable
        Pair<ExoTrackSelection.Definition, Integer> selectedAudio =
                selectAudioTrack(
                        mappedTrackInfo,
                        rendererFormatSupports,
                        rendererMixedMimeTypeAdaptationSupports,
                        params);
        if (selectedAudio != null) {
            definitions[selectedAudio.second] = selectedAudio.first;
        }

        @Nullable
        String selectedAudioLanguage =
                selectedAudio == null
                        ? null
                        : selectedAudio.first.group.getFormat(selectedAudio.first.tracks[0]).language;
        @Nullable
        Pair<ExoTrackSelection.Definition, Integer> selectedText =
                selectTextTrack(mappedTrackInfo, rendererFormatSupports, params, selectedAudioLanguage);
        if (selectedText != null) {
            definitions[selectedText.second] = selectedText.first;
        }

        for (int i = 0; i < rendererCount; i++) {
            int trackType = mappedTrackInfo.getRendererType(i);
            if (trackType != C.TRACK_TYPE_VIDEO
                    && trackType != C.TRACK_TYPE_AUDIO
                    && trackType != C.TRACK_TYPE_TEXT
                    && trackType != C.TRACK_TYPE_IMAGE) {
                definitions[i] =
                        selectOtherTrack(
                                trackType, mappedTrackInfo.getTrackGroups(i), rendererFormatSupports[i], params);
            }
        }
        return definitions;
    }

    private <T extends TrackInfo<T>> ArrayList<Pair<ExoTrackSelection.Definition, Integer>> selectVideoTracks(
            MappedTrackInfo mappedTrackInfo,
            int[][][] rendererFormatSupports,
            Parameters params,
            TrackInfo.Factory<T> trackInfoFactory,
            Comparator<List<T>> selectionComparator
    ) {
        if (params.audioOffloadPreferences.audioOffloadMode == AUDIO_OFFLOAD_MODE_REQUIRED) {
            return null;
        }
        @C.TrackType int trackType = C.TRACK_TYPE_VIDEO;
        ArrayList<List<T>> possibleSelections;
        ArrayList<ArrayList<List<T>>> possibleSelectionsPerRenderer = new ArrayList<>();
        int rendererCount = mappedTrackInfo.getRendererCount();
        for (int rendererIndex = 0; rendererIndex < rendererCount; rendererIndex++) {
            possibleSelections = new ArrayList<>();
            if (trackType == mappedTrackInfo.getRendererType(rendererIndex)) {
                TrackGroupArray groups = mappedTrackInfo.getTrackGroups(rendererIndex);
                for (int groupIndex = 0; groupIndex < groups.length; groupIndex++) {
                    TrackGroup trackGroup = groups.get(groupIndex);
                    @RendererCapabilities.Capabilities int[] groupSupport = rendererFormatSupports[rendererIndex][groupIndex];
                    List<T> trackInfos = trackInfoFactory.create(rendererIndex, trackGroup, groupSupport);
                    boolean[] usedTrackInSelection = new boolean[trackGroup.length];
                    for (int trackIndex = 0; trackIndex < trackGroup.length; trackIndex++) {
                        T trackInfo = trackInfos.get(trackIndex);
                        @SelectionEligibility int eligibility = trackInfo.getSelectionEligibility();
                        if (usedTrackInSelection[trackIndex] || eligibility == SELECTION_ELIGIBILITY_NO) {
                            continue;
                        }
                        List<T> selection;
                        if (eligibility == SELECTION_ELIGIBILITY_FIXED) {
                            selection = ImmutableList.of(trackInfo);
                        } else {
                            selection = new ArrayList<>();
                            selection.add(trackInfo);
                            for (int i = trackIndex + 1; i < trackGroup.length; i++) {
                                T otherTrackInfo = trackInfos.get(i);
                                if (otherTrackInfo.getSelectionEligibility() == SELECTION_ELIGIBILITY_ADAPTIVE) {
                                    if (trackInfo.isCompatibleForAdaptationWith(otherTrackInfo)) {
                                        selection.add(otherTrackInfo);
                                        usedTrackInSelection[i] = true;
                                    }
                                }
                            }
                        }
                        possibleSelections.add(selection);
                    }
                }
            }
            possibleSelectionsPerRenderer.add(possibleSelections);
        }
        if (possibleSelectionsPerRenderer.isEmpty()) {
            return null;
        }
        ArrayList<Pair<ExoTrackSelection.Definition, Integer>> selectedVideos = new ArrayList<>();
        for (ArrayList<List<T>> selections: possibleSelectionsPerRenderer) {
            if (selections.isEmpty()) continue;
            List<T> bestSelection = max(selections, selectionComparator);
            int[] trackIndices = new int[bestSelection.size()];
            for (int i = 0; i < bestSelection.size(); i++) {
                trackIndices[i] = bestSelection.get(i).trackIndex;
            }
            T firstTrackInfo = bestSelection.get(0);
            selectedVideos.add(Pair.create(
                    new ExoTrackSelection.Definition(firstTrackInfo.trackGroup, trackIndices),
                    firstTrackInfo.rendererIndex)
            );
        }
        return selectedVideos;
    }
}

Below is my playerManager code.

public class PlayerManager {
    private final Context context;
    private ExoPlayer player;
    private final SurfaceViewManager surfaceViewManager;
    private final ArrayList<String> cameraNames = new ArrayList<>();
    private static final String TAG = "HELLO";

    public PlayerManager(Context context, SurfaceViewManager surfaceViewManager) {
        this.context = context;
        this.surfaceViewManager = surfaceViewManager;
    }

    @OptIn(markerClass = UnstableApi.class)
    public ExoPlayer initializePlayer(int numOfViews, Map<String, String> replayData){
        for (Map.Entry<String, String> entry : replayData.entrySet()) {
            cameraNames.add(entry.getKey());
        }

        DashMediaSource dashMediaSource = new DashMediaSource.Factory(new DefaultDataSource.Factory(context))
                .createMediaSource(MediaItem.fromUri(Uri.parse("http://192.168.0.113:8009/replay_emo_goal3_manifest.mpd")));

        player = new ExoPlayer.Builder(context)
                .setRenderersFactory(new CustomRenderersFactory(context, numOfViews))
                .setTrackSelector(new CustomTrackSelector(context))
                .build();
        player.setMediaSource(dashMediaSource);
        player.prepare();
        player.setRepeatMode(Player.REPEAT_MODE_ALL);
        player.setPlayWhenReady(true);
        return player;
    }

    public void switchRendererTrack(){
        logActiveAdaptationSets();
        switchRendererTrackGroup(0, 0);
    }

    @OptIn(markerClass = UnstableApi.class)
    public ArrayList<Renderer> setupRenderers(ExoPlayer player) {
        ArrayList<Renderer> videoRenderers = new ArrayList<>();
        for (int i = 0; i < player.getRendererCount(); i++) {
            if (player.getRendererType(i) == C.TRACK_TYPE_VIDEO) {
                videoRenderers.add(player.getRenderer(i));
            }
        }
        surfaceViewManager.attachRenderersToSurfaces(player, videoRenderers);
        surfaceViewManager.attachLabelsToTextViews(cameraNames);
        return  videoRenderers;
    }

    @OptIn(markerClass = UnstableApi.class)
    public void switchRendererTrackGroup(int rendererIndex, int trackGroupIndex) {
        DefaultTrackSelector trackSelector = (DefaultTrackSelector) player.getTrackSelector();
        if (trackSelector != null) {
            DefaultTrackSelector.Parameters.Builder parametersBuilder = trackSelector.buildUponParameters();
            DefaultTrackSelector.MappedTrackInfo mappedTrackInfo = trackSelector.getCurrentMappedTrackInfo();

            if (mappedTrackInfo == null || rendererIndex >= mappedTrackInfo.getRendererCount()) {
                Log.w(TAG, "Invalid renderer index or track info is unavailable.");
                return;
            }

            TrackGroupArray trackGroups = mappedTrackInfo.getTrackGroups(rendererIndex);

            if (trackGroupIndex < 0 || trackGroupIndex >= trackGroups.length) {
                Log.w(TAG, "Invalid trackGroupIndex: " + trackGroupIndex);
                return;
            }

            parametersBuilder.clearOverridesOfType(rendererIndex)
                    .setSelectionOverride(
                            rendererIndex,
                            trackGroups,
                            new DefaultTrackSelector.SelectionOverride(trackGroupIndex, 0)
                    );

            // Apply the new parameters to the TrackSelector
            trackSelector.setParameters(parametersBuilder.build());

            Log.d(TAG, "Switched to TrackGroup " + trackGroupIndex + " for renderer " + rendererIndex);
        }
    }

    @OptIn(markerClass = UnstableApi.class)
    public void logActiveAdaptationSets() {
        if (player == null) {
            Log.w(TAG, "Player is not initialized.");
            return;
        }

        for (int i = 0; i < player.getRendererCount(); i++) {
            if (player.getRendererType(i) == C.TRACK_TYPE_VIDEO) {
                TrackSelection trackSelection = player.getCurrentTrackSelections().get(i);

                if (trackSelection != null) {
                    TrackGroup trackGroup = trackSelection.getTrackGroup();
                    String adaptationSetInfo = "Renderer " + i + ": Adaptation Set Index " + trackGroup.getFormat(0).id;
                    Log.d(TAG, adaptationSetInfo);
                } else {
                    Log.d(TAG, "Renderer " + i + ": No active adaptation set.");
                }
            }
        }
    }

    public void releasePlayer() {
        if (player != null) {
            player.stop();
            player.release();
            player = null;
        }
    }
}

Currently if there are more video trackgroups than the number of video renderers, all the extra trackgroups are assigned to the firstRenderer alone. ie, when I log mappedTrackInfo.getTrackGroups(rendererIndex) i get

Renderer 1 Support Track groups 1,5 and 6 Renderer 2 Supports Track Group 2 Renderer 3 Supports Track Group 3 Renderer 4 Supports Track Group 4

With the current implementation I am able to switch the trackGroup between 1,5 and 6 for renderer one alone but I am not able to set trackGroup 5 or 6 to renderer 2, 3 and 4.

Is there a way such that all the renderers support all the track groups so that I can switch between them or is there any way I can switch the trackGroup of a renderer though it does not have the trackGroup assigned to it.

Also, setSelectionOverride is deprecated, but when I use addOverride and perform the switch, all my other renderers stop playing. Is there a way to prevent this from happening with addOverride.

Below is my current addOverride implementation

parametersBuilder.clearOverridesOfType(rendererIndex)
                    .addOverride(new TrackSelectionOverride(trackGroups.get(trackGroupIndex), 0));

[icbaker]

I'm afraid your question is asking about a relatively niche use case, and we don't have the capacity to parse and understand all the custom code you've provided.

Currently if there are more video trackgroups than the number of video renderers, all the extra trackgroups are assigned to the firstRenderer alone. ie, when I log mappedTrackInfo.getTrackGroups(rendererIndex) i get

Renderer 1 Support Track groups 1,5 and 6 Renderer 2 Supports Track Group 2 Renderer 3 Supports Track Group 3 Renderer 4 Supports Track Group 4

In your case, I would expect all renderers to indicate support for all the track groups - so I'm not sure how or why you're limiting the support to a specific mapping - but I recommend you spend some time on understanding this part, because it doesn't seem right.