💭 How to support pagination with this?

[Meet-Miyani]

🤔 How to support pagination with this?

I am trying to fetch the videos but with pagination. And I already setup required things, but not sure how to fit it in the end. As I find this code a bit complex architecture to understand. I don't have much experience with flows.

What I did for paging is:

• [x] VideoDataSource
• [X] PagerPagingAdapter

Now problem here is, in the code there are many things which are shared in a flow, which extends ViewResult, I am referring to the VideoPagerViewModel, according to the code I am able to use Flow<List> but according to my implementation I will be getting Flow<PagingData> this is where the issue is happening, I can't make the PagingData extend ViewResult directly also. So how can I achieve the pagination in this? Because in ShortsFragment I can see that code fetches all the videos at once.

ShortsFragment (🗝️ Already Present)

.
.
val states = viewModel.states
            .onEach { state ->
                // Await the list submission so that the adapter list is in sync with state.videoData
                adapter.awaitList(state.videoData)
                }
.
.

VideoPagerViewModel (🗝️ Already Present)

internal class VideoPagerViewModel(
    private val repository: VideoDataRepository,
    private val appPlayerFactory: AppPlayer.Factory,
    private val handle: PlayerSavedStateHandle,
    initialState: ViewState,
) : MviViewModel<ViewEvent, ViewResult, ViewState, ViewEffect>(initialState) {

    override fun onStart() {
        processEvent(LoadVideoDataEvent)
    }

    override fun Flow<ViewEvent>.toResults(): Flow<ViewResult> {
        // MVI boilerplate
        return merge(
            filterIsInstance<LoadVideoDataEvent>().toLoadVideoDataResults(),
            filterIsInstance<PlayerLifecycleEvent>().toPlayerLifecycleResults(),
            filterIsInstance<TappedPlayerEvent>().toTappedPlayerResults(),
            filterIsInstance<OnPageSettledEvent>().toPageSettledResults(),
            filterIsInstance<PauseVideoEvent>().toPauseVideoResults()
        )
    }

    private fun Flow<LoadVideoDataEvent>.toLoadVideoDataResults(): Flow<ViewResult> {
        return flatMapLatest { repository.videoData() }
            .map { videoData ->
                val appPlayer = states.value.appPlayer
                // If the player exists, it should be updated with the latest video data that came in
                appPlayer?.setUpWith(videoData, handle.get())
                // Capture any updated index so UI page state can stay in sync. For example, a video
                // may have been added to the page before the currently active one. That means the
                // the current video/page index will have changed
                val index = appPlayer?.currentPlayerState?.currentMediaItemIndex ?: 0
                LoadVideoDataResult(videoData, index)
            }
    }
.
.
.
}

VideoDataSource (🆕 Added)

class VideoDataSource(private val videoDao: VideosDao?) : PagingSource<Int, VideoData>() {

    override fun getRefreshKey(state: PagingState<Int, VideoData>): Int? {
        return state.anchorPosition?.let { anchorPosition ->
            val anchorPage = state.closestPageToPosition(anchorPosition)
            anchorPage?.prevKey?.plus(1) ?: anchorPage?.nextKey?.minus(1)
        }
    }

    override suspend fun load(params: LoadParams<Int>): LoadResult<Int, VideoData> {
        val page = params.key ?: 0

        return try {
            val videos = videoDao?.getPaginatedShorts(params.loadSize, page * params.loadSize)
                ?: emptyList()
            LoadResult.Page(
                data = videos,
                prevKey = if (page == 0) null else page - 1,
                nextKey = if (videos.isEmpty()) null else page + 1
            )
        } catch (e: Exception) {
            LoadResult.Error(e)
        }
    }
}

PagerPagingAdapter (🆕 Added)

internal class PagerPagingAdapter(private val imageLoader: ImageLoader) :
    PagingDataAdapter<VideoData, PageViewHolder>(VideoDataDiffCallback) {

    private var recyclerView: RecyclerView? = null

    // Extra buffer capacity so that emissions can be sent outside a coroutine
    private val clicks = MutableSharedFlow<Unit>(extraBufferCapacity = 1)

    fun clicks() = clicks.asSharedFlow()

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): PageViewHolder {
        return LayoutInflater.from(parent.context)
            .let { inflater -> ShortsPageItemBinding.inflate(inflater, parent, false) }
            .let { binding ->
                PageViewHolder(binding, imageLoader) { clicks.tryEmit(Unit) }
            }
    }

    override fun onBindViewHolder(holder: PageViewHolder, position: Int) {
        getItem(position)?.let(holder::bind)
    }

    override fun onAttachedToRecyclerView(recyclerView: RecyclerView) {
        this.recyclerView = recyclerView
    }

    override fun onDetachedFromRecyclerView(recyclerView: RecyclerView) {
        this.recyclerView = null
    }

    /**
     * Attach [appPlayerView] to the ViewHolder at [position]. The player won't actually be visible in
     * the UI until [showPlayerFor] is also called.
     */
    suspend fun attachPlayerView(appPlayerView: AppPlayerView, position: Int) {
        awaitViewHolder(position).attach(appPlayerView)
    }

    // Hides the video preview image when the player is ready to be shown.
    suspend fun showPlayerFor(position: Int) {
        awaitViewHolder(position).hidePreviewImage()
    }

    suspend fun renderEffect(position: Int, effect: PageEffect) {
        awaitViewHolder(position).renderEffect(effect)
    }

    /**
     * The ViewHolder at [position] isn't always immediately available. In those cases, wait for
     * the RecyclerView to be laid out and re-query that ViewHolder.
     */
    private suspend fun awaitViewHolder(position: Int): PageViewHolder {
        if (itemCount == 0) error("Tried to get ViewHolder at position $position, but the list was empty")

        var viewHolder: PageViewHolder?

        do {
            viewHolder = recyclerView?.findViewHolderForAdapterPosition(position) as? PageViewHolder
        } while (currentCoroutineContext().isActive && viewHolder == null && recyclerView?.awaitNextLayout() == Unit)

        return requireNotNull(viewHolder)
    }

    private object VideoDataDiffCallback : DiffUtil.ItemCallback<VideoData>() {
        override fun areItemsTheSame(oldItem: VideoData, newItem: VideoData): Boolean {
            return oldItem.id == newItem.id
        }

        override fun areContentsTheSame(oldItem: VideoData, newItem: VideoData): Boolean {
            return oldItem == newItem
        }
    }
}

[Meet-Miyani]

@nihk can you please help here?

[nihk]

hi @Meet-Miyani. i'm not familiar with the androidx.paging library, unfortunately. in my opinion, it's a library one should avoid using unless your use case is handling hundreds of thousands of records in a database table.

but to your question: VideoViewModel.toLoadVideoDataResults() is where the video data is loaded from. you could change object LoadVideoDataEvent to instead be something like data class LoadVideoDataEvent(val page: Int = 1) and forward the page value to your VideoDataRepository implementation.

upstream the UI can call viewModel.processEvent(LoadVideoDataEvent(page = ...)) once it reaches the last video in the list.

in VideoViewModel.reduce() for LoadVideoDataResult you'll want to append the videoData to the current state's videoData instead of outright replacing it.

[Meet-Miyani]

I see, seems a bit difficult for me @nihk But I'll give a sure try! Thanks for considering my issue ❤️ Actually yes, I have to use the paging library as I have thousands of videos in my db, and handling all at once take some time like 3 to 4 secs for the viewpager to setup. But I will try your approach

Again thanks a lot. If you don't mind can I reach back to you in case I couldn't figure it out?

[nihk]

no problem. let me know how it goes!

[Meet-Miyani]

Hey @nihk , seems like it did work, but still I am messing up something, maybe with concatenation of old and new data.

I've taken pageSize as 5 for checking it faster. Will increase it later

Let me show you the code changes,

VideoDataRepository

interface VideoDataRepository {
    fun videoData(page: Int): Flow<List<VideoData>>
}

ShortVideoDataRepository

class ShortVideoDataRepository(private val context: Context) : VideoDataRepository {
    override fun videoData(page: Int): Flow<List<VideoData>> {
        return ViewTubeDatabase.getInstance(context).shortsDao().getPaginatedShorts(5, page)
            .map { shorts ->
                shorts.map { short ->
                    VideoData(
                        short.id,
                        Uri.parse(short.uri),
                        short.path,
                        short.aspectRatio,
                        short.duration
                    )
                }
            }
    }
}

ShortsFragment

override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)
        val binding = FragmentShortsBinding.bind(view)
        // This single player view instance gets attached to the ViewHolder of the active ViewPager page
        val appPlayerView = appPlayerViewFactory.create(view.context)

        val adapter = PagerAdapter(imageLoader)
        binding.shortPager.adapter = adapter
        binding.shortPager.offscreenPageLimit = 1 // Preload neighbouring page image previews

        val states = viewModel.states
            .onEach { state ->
                .
                .
                .                
                 // Restore any saved page state from process recreation and configuration changes.
                // Guarded by an isIdle check so that state emissions mid-swipe or during page change
                // animations are ignored. There would have a jarring page-change effect without that.
                if (binding.shortPager.isIdle) {
                    binding.shortPager.setCurrentItem(state.page, false)
                    state.videoData?.let {
                        if (state.page > it.lastIndex - 2) {
                            val page = it.size / 5
                            viewModel.processEvent(LoadVideoDataEvent(page)) // -- need help here as well
                        }
                    }
                }
.
.
.
    }

VideoPagerViewModel

internal class VideoPagerViewModel(
    private val repository: VideoDataRepository,
    private val appPlayerFactory: AppPlayer.Factory,
    private val handle: PlayerSavedStateHandle,
    initialState: ViewState,
) : MviViewModel<ViewEvent, ViewResult, ViewState, ViewEffect>(initialState) {

    override fun onStart() {
        processEvent(LoadVideoDataEvent())
    }

    override fun Flow<ViewEvent>.toResults(): Flow<ViewResult> {
        // MVI boilerplate
        return merge(
            filterIsInstance<LoadVideoDataEvent>().toLoadVideoDataResults(),
            filterIsInstance<PlayerLifecycleEvent>().toPlayerLifecycleResults(),
            filterIsInstance<TappedPlayerEvent>().toTappedPlayerResults(),
            filterIsInstance<OnPageSettledEvent>().toPageSettledResults(),
            filterIsInstance<PauseVideoEvent>().toPauseVideoResults()
        )
    }

    private fun Flow<LoadVideoDataEvent>.toLoadVideoDataResults(): Flow<ViewResult> {
        return flatMapLatest { repository.videoData(it.page) }  // <------  sent the page here
            .map { videoData ->
                val appPlayer = states.value.appPlayer
                // If the player exists, it should be updated with the latest video data that came in
                appPlayer?.setUpWith(videoData, handle.get())
                // Capture any updated index so UI page state can stay in sync. For example, a video
                // may have been added to the page before the currently active one. That means the
                // the current video/page index will have changed
                val index = appPlayer?.currentPlayerState?.currentMediaItemIndex ?: 0
                LoadVideoDataResult(videoData, index)
            }
    }

    override fun ViewResult.reduce(state: ViewState): ViewState {
        // MVI reducer boilerplate
        return when (this) {
            is LoadVideoDataResult -> {  // ---- need help here, maybe problem with joining the 2 lists.
                val videos = arrayListOf<VideoData>()
                state.videoData?.let { videos.addAll(it) }
                videos.addAll(videoData)
                state.copy(
                    videoData = videos,
                    page = currentMediaItemIndex
                )
            }

            is CreatePlayerResult -> state.copy(appPlayer = appPlayer)
            is TearDownPlayerResult -> state.copy(appPlayer = null)
            is OnNewPageSettledResult -> state.copy(page = page, showPlayer = false)
            is OnPlayerRenderingResult -> state.copy(showPlayer = true)
            is AttachPlayerToViewResult -> state.copy(attachPlayer = doAttach)
            else -> state
        }
    }
}

[nihk]

i think in toLoadVideoDataResults is where you should be combining the videoData, not reduce(), e.g.

    private fun Flow<LoadVideoDataEvent>.toLoadVideoDataResults(): Flow<ViewResult> {
        return flatMapLatest { repository.videoData(it.page) }  // <------  sent the page here
            .map { videoData ->
                val combinedVideoData = states.value.videoData.orEmpty() + videoData
                val appPlayer = states.value.appPlayer
                // If the player exists, it should be updated with the latest video data that came in
                appPlayer?.setUpWith(combinedVideoData , handle.get())
                // Capture any updated index so UI page state can stay in sync. For example, a video
                // may have been added to the page before the currently active one. That means the
                // the current video/page index will have changed
                val index = appPlayer?.currentPlayerState?.currentMediaItemIndex ?: 0
                LoadVideoDataResult(combinedVideoData , index)
            }
    }

[nihk]

i'd forgotten about the appPlayer?.setUpWith part

[nihk]

                    state.videoData?.let {
                        if (state.page > it.lastIndex - 2) {
                            val page = it.size / 5
                            viewModel.processEvent(LoadVideoDataEvent(page)) // -- need help here as well
                        }
                    }

this part shouldn't be in the viewModel.states.onEach block; you should send events as a side effect of the page changes.

also in hindsight, avoid duplicating the variable name page for the individual page and the broader cursor page; it'll get confusing mixing up the two different uses. maybe something like cursorPage for the latter.

adding something like this will probably work:

    private fun Flow<LoadVideoDataEvent>.toLoadVideoDataResults(): Flow<ViewResult> {
        return filter { event -> states.value.videoData == null || event.page == states.value.videoData.orEmpty().lastIndex }
         .flatMapLatest { event -> repository.videoData(states.value.cursorPage + 1) } // cursorPage is new, you can track it
// you'll also want to update the `cursorPage` value via LoadVideoDataResult

[nihk]

might want to use flatMapConcat instead of flatMapLatest too, to avoid accidentally discarding downstream event processing

[nihk]

another approach instead of the above is to manage the current page state within your VideoDataRepository implementation, and simply call something like VideoDataRepository.nextPage() when desired conditions are met in toPageSettledResults. this'd work because VideoDataRepository.videoData() returns a stream and not a 1-shot function. VideoDataRepository could even own all the logic to manage concatenating videoData if you wanted.

[Meet-Miyani]

I see, sorry to come up again. I am really naive with the flows and the architecture you implemented in this. Thanks again for helping this much 🙏🏽

1. I removed the logic of changing page separate as you instructed.
2. Moved the code from reduce to toLoadVideoDataResults

See, here

binding.shortPager.registerOnPageChangeCallback(object : ViewPager2.OnPageChangeCallback() {
            override fun onPageSelected(position: Int) {
                super.onPageSelected(position)
                if (position > adapter.currentList.lastIndex - 1) {
                    val page = adapter.currentList.size / 5
                    viewModel.processEvent(LoadVideoDataEvent(page))
                }
            }
        })

private fun Flow<LoadVideoDataEvent>.toLoadVideoDataResults(): Flow<ViewResult> {
        return filter { event ->
            Log.d("Shorts_Fragment", "states.value.videoData: ${states.value.videoData}")
            Log.d("Shorts_Fragment", "event.page: ${event.page}")
            Log.d("Shorts_Fragment", "states.value.videoData.orEmpty().lastIndex: ${states.value.videoData.orEmpty().lastIndex}")
            states.value.videoData == null || event.page == states.value.videoData.orEmpty().lastIndex
        }.flatMapLatest { event ->
            Log.d("Shorts_Fragment", "states.value.cursorPage + 1: ${states.value.cursorPage + 1}")
                repository.videoData(states.value.cursorPage + 1)
            }
            .map { videoData ->
                val combinedVideoData = states.value.videoData.orEmpty() + videoData
                val appPlayer = states.value.appPlayer
                // If the player exists, it should be updated with the latest video data that came in
                appPlayer?.setUpWith(combinedVideoData, handle.get())
                // Capture any updated index so UI page state can stay in sync. For example, a video
                // may have been added to the page before the currently active one. That means the
                // the current video/page index will have changed
                val index = appPlayer?.currentPlayerState?.currentMediaItemIndex ?: 0
                LoadVideoDataResult(videoData, index)
            }
    }

Check the logs, seems like it is able to load the next page results, I guess. Logs:

states.value.videoData: null

event.page: 0

states.value.videoData.orEmpty().lastIndex: -1

states.value.cursorPage + 1: 1

states.value.videoData: [VideoData(id=1000177944, mediaUri=content://media/external/video/media/1000177944, previewImageUri=/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Video/VID-20240228-WA0038.mp4, aspectRatio=0.5625, videoDuration=15070), VideoData(id=1000177778, mediaUri=content://media/external/video/media/1000177778, previewImageUri=/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Animated Gifs/VID-20240228-WA0013.mp4, aspectRatio=1.7931035, videoDuration=4230), VideoData(id=1000173921, mediaUri=content://media/external/video/media/1000173921, previewImageUri=/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Animated Gifs/VID-20240211-WA0038.mp4, aspectRatio=1.3333334, videoDuration=2100), VideoData(id=1000177647, mediaUri=content://media/external/video/media/1000177647, previewImageUri=/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Animated Gifs/VID-20240228-WA0001.mp4, aspectRatio=1.0, videoDuration=980), VideoData(id=1000177558, mediaUri=content://media/external/video/media/1000177558, previewImageUri=/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Video/VID-20240227-WA0007.mp4, aspectRatio=0.55833334, videoDuration=4565)]

event.page: 1

states.value.videoData.orEmpty().lastIndex: 4

[Meet-Miyani]

Hey @nihk, please help here brother!

[Meet-Miyani]

Hello @nihk is there any update? If you have some free time, I will appreciate your help here, Thanks!