Support a preview seekbar for showing previews when seeking

[Luna712]

Describe your suggested feature

It would be nice if the player supported a preview seekbar, for showing thumbnail previews while you are seeking. Something like https://github.com/rubensousa/PreviewSeekBar which has examples for media3 implementation as well. It is also fairly maintained.

If not the library, there may also be a solution using something similar to this: https://stackoverflow.com/a/62006472

Thumbnail sprite could probably be created as a temporary file using something like ffmpeg for online videos and perhaps something like MediaMetadataRetriever for local videos. It could be generated in the loading step of a video, or even after while the video is playing or on first attempted seek and only stored while the video is playing, then removed. You could also have it save under a single file name and overwrite when a new one is generated so you only ever have one thumbnail sprite file generated and to look at.

In my opinion this would make the built in cloudstream player so much better and alot more friendly usage, and consistently with popular streaming services that support something like this, YouTube, Netflix, Google TV, Etc...

This is just an idea and I understand it might be complex, but I do believe it would be a fairly worthwhile feature improvement to the media player. If this is not something that will even be consider then I guess I understand, but it does seem to me like something at least feasible.

My technical knowledge of this is limited at best so if my theory here is totally not feasible then I apologize as well.

Other details

No response

Acknowledgements

• [X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
• [X] I have written a short but informative title.
• [X] I will fill out all of the requested information in this form.

[recloudstream[bot]]

Hello Luna712. Please do not report any provider bugs here. This repository does not contain any providers. Please find the appropriate repository and report your issue there or join the discord.

Found provider name: IMoviehd

[Blatzar]

I am well aware of both that exact stackoverflow question and that library as I have investigated this before. Actually doing the UI is not that difficult, getting proper thumbnails is the harder part.

MediaMetadataRetriever actually looks promising, but downloaded videos is pretty low priority (less used + no buffering). Ffmpeg is pretty much the most viable method to get thumbnails in the least shitty way, but that itself requires a lot of work to implement. It is simply not worth it time wise compared to other requests, even though it is a good request.

Developing all boils down to usefulness/time and this is much time and not that high usefulness :pensive:

[Luna712]

I understand it might be quite complex, I probably wouldn't know how to implement it into the existing app code, but I can try to learn the codebase and implement something like this if it is something you'd actually be interested in having merged in?

[Luna712]

This is a very rudimentary example of potential implementation, for local videos, it does generate a basic (not complete or finished by any means) thumbnail sprite, with ffmpeg basic support planned (unfinished), and is not with CloudStream code base but to just output the sprite in a basic (seperate) app, for example implementation for potential starting. If this actually seems like something interesting, I might try and go further for a PR here for full complete implementation, but for now this is just a PoC example implementation (note code style formatting is not great either):

import android.os.Environment
import android.content.Context
import android.widget.ImageView

import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.media.MediaMetadataRetriever
import android.os.AsyncTask
import android.util.Log
// TODO: This is an archived library, use something else
// import com.arthenica.mobileffmpeg.Config
// import com.arthenica.mobileffmpeg.FFmpeg
import java.io.File
import java.io.FileOutputStream
import java.io.IOException
import java.net.URL

         val downloadsDir = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS)

         val inputVideoPath = File(downloadsDir, "Movies/Movie.mp4").toString()
         val outputSpritePath = File(downloadsDir, "Movies/sprite.png").toString()

        generateThumbnailSprite(inputVideoPath, outputSpritePath) {
            // This callback is executed when the thumbnail sprite generation is complete

            val thumbnailImageView = findViewById<ImageView>(R.id.thumbnailImageView)
            val thumbnailBitmap = BitmapFactory.decodeFile(outputSpritePath)

            runOnUiThread {
                thumbnailImageView.setImageBitmap(thumbnailBitmap)
            }
    }

fun generateThumbnailSprite(inputPath: String, outputSpritePath: String, callback: () -> Unit) {
    val isOnlineVideo = inputPath.startsWith("http")

    if (isOnlineVideo) {
        // TODO
      /*  // If it's an online video, use FFmpeg to generate the sprite
        val cmd = arrayOf(
            "-i", inputPath,
            "-vf", "thumbnail=10,setpts=N/10*(TB)",
            "-frames:v", "10",
            outputSpritePath
        )

        val result = FFmpeg.execute(cmd)

        if (result == Config.RETURN_CODE_SUCCESS) {
            callback()
        } else {
            Log.e("Thumbnail Generation", "FFmpeg execution failed")
        }*/
    } else {
        // If it's a local video, use MediaMetadataRetriever to generate the sprite
        // TODO: this is very basic, improve number of frames, intervals, and layout for sprite to be used properly
        AsyncTask.execute {
            val retriever = MediaMetadataRetriever()
            retriever.setDataSource(inputPath)

            val bitmaps = ArrayList<Bitmap>()
            val frameInterval = 30000000L // 30 seconds in microseconds
            for (i in 0 until 10) {
                val timeUs = i * frameInterval
                val bitmap = retriever.getFrameAtTime(timeUs, MediaMetadataRetriever.OPTION_CLOSEST_SYNC)
                if (bitmap != null) {
                    bitmaps.add(bitmap)
                }
            }

            val spriteBitmap = createThumbnailSprite(bitmaps)
            saveBitmapToFile(spriteBitmap, outputSpritePath)

            retriever.release()
            callback()
        }
    }
}

fun createThumbnailSprite(bitmaps: List<Bitmap>): Bitmap {
    // Combine the bitmaps into a single sprite
    // TODO: change/improve layout

    val width = bitmaps[0].width
    val height = bitmaps[0].height * bitmaps.size

    val spriteBitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
    val canvas = android.graphics.Canvas(spriteBitmap)

    var top = 0
    for (bitmap in bitmaps) {
        canvas.drawBitmap(bitmap, 0f, top.toFloat(), null)
        top += bitmap.height
    }

    return spriteBitmap
}

fun saveBitmapToFile(bitmap: Bitmap, filePath: String) {
    val file = File(filePath)
    try {
        val outputStream = FileOutputStream(file)
        bitmap.compress(Bitmap.CompressFormat.PNG, 100, outputStream)
        outputStream.flush()
        outputStream.close()
    } catch (e: IOException) {
        Log.e("Thumbnail Generation", "Error saving sprite bitmap to file")
    }
}

[Luna712]

Here is a much nicer version that generates a fairly decent file for local videos, still need to do the ffmpeg work though:

import android.graphics.Bitmap
import android.graphics.Canvas
import android.media.MediaMetadataRetriever
import java.io.File
import java.io.FileOutputStream

// TODO: This is an archived lib
// import com.arthenica.mobileffmpeg.Config
// import com.arthenica.mobileffmpeg.FFmpeg

internal interface ThumbnailSpriteCallback {
    fun onThumbnailSpriteGenerated(spritePath: String)
    fun onThumbnailSpriteGenerationError(error: Exception)
}

internal class ThumbnailSpriteGenerator(
    private val inputVideoPath: String,
    private val outputSpritePath: String,
    private val callback: ThumbnailSpriteCallback
) {

    private val maxLines = 6
    private val maxColumns = 10

    private val thumbnailHeight = 100
    private val thumbnailWidth = 150

    private val oneMinute = 60000L

   internal fun generateThumbnailSprite() {
        val videoDuration = getVideoDuration(inputVideoPath)

        if (videoDuration <= 0) {
            callback.onThumbnailSpriteGenerationError(Exception("Invalid video duration"))
            return
        }

        val thumbnailList = ArrayList<Bitmap>()

        for (timeInMillis in 0 until videoDuration step oneMinute) {
            val thumbnail = generateThumbnail(inputVideoPath, timeInMillis)
            if (thumbnail != null) {
                thumbnailList.add(thumbnail)
            }
        }

       try {
            val spriteBitmap = createSpriteBitmap(thumbnailList)
            saveSpriteToFile(spriteBitmap, outputSpritePath)
            callback.onThumbnailSpriteGenerated(outputSpritePath)
        } catch (e: Exception) {
            e.printStackTrace()
            callback.onThumbnailSpriteGenerationError(e)
        }
    }

    private fun generateThumbnail(videoPath: String, timeInMillis: Long): Bitmap? {
        val retriever = MediaMetadataRetriever()

        return try {
            val isOnlineFile = videoPath.startsWith("http://") || videoPath.startsWith("https://")
            if (isOnlineFile) {
                // Implement FFmpeg for online video thumbnail generation
                null
            } else {
                retriever.setDataSource(videoPath)

                // Calculate the frame time based on timeInMillis
                val frameTimeMicros = timeInMillis * 1000

                // Retrieve the frame at the specified time
                val frameBitmap = retriever.getFrameAtTime(frameTimeMicros, MediaMetadataRetriever.OPTION_CLOSEST_SYNC)

                if (frameBitmap != null) {
                    var scaledBitmap = Bitmap.createScaledBitmap(frameBitmap, thumbnailWidth, thumbnailHeight, false)

                    // Release the MediaMetadataRetriever
                    retriever.release()

                    scaledBitmap
                } else {
                    // Release the MediaMetadataRetriever
                    retriever.release()
                    null
                }
            }
        } catch (e: Exception) {
            e.printStackTrace()
            callback.onThumbnailSpriteGenerationError(e)
            null
        }
    }

    private fun createSpriteBitmap(thumbnails: List<Bitmap>): Bitmap {
        val spriteWidth = thumbnails[0].width * maxColumns
        val spriteHeight = thumbnails[0].height * maxLines
        val sprite = Bitmap.createBitmap(spriteWidth, spriteHeight, Bitmap.Config.ARGB_8888)
        val canvas = Canvas(sprite)

        for (i in thumbnails.indices) {
            val x = i % maxColumns * thumbnails[0].width
            val y = i / maxColumns * thumbnails[0].height
            canvas.drawBitmap(thumbnails[i], x.toFloat(), y.toFloat(), null)
        }

        return sprite
    }

    private fun saveSpriteToFile(bitmap: Bitmap, filePath: String) {
        val file = File(filePath)
        val outputStream = FileOutputStream(file)
        bitmap.compress(Bitmap.CompressFormat.PNG, 100, outputStream)
        outputStream.flush()
        outputStream.close()
    }

    private fun getVideoDuration(videoPath: String): Long {
        val retriever = MediaMetadataRetriever()
        retriever.setDataSource(videoPath)

        val durationString = retriever.extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION)

        return durationString?.toLong() ?: 0
    }
}

And usage:

import android.os.Environment
import android.os.AsyncTask
import android.util.Log
import java.util.Random
import java.io.File

import android.widget.VideoView

public class MainActivity : AppCompatActivity() {

    private lateinit var videoView: VideoView

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        val downloadsDir = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS)

        // Just use a random file from the Movies directory for now, just for testing
        val moviesDir = File(downloadsDir, "Movies")

        val mp4Files = moviesDir.listFiles { file ->
            file.isFile && file.name.endsWith(".mp4", ignoreCase = true)
        }

        val random = Random()
        val randomIndex = random.nextInt(mp4Files.size)
        val randomMp4File = mp4Files[randomIndex]

        val inputVideoPath = randomMp4File.toString()

        val outputSpritePath = File(moviesDir, "sprite.png").toString()

                videoView = findViewById(R.id.videoView)
        videoView.setVideoPath(inputVideoPath)
        videoView.start()

        val spriteGenerator = ThumbnailSpriteGenerator(inputVideoPath, outputSpritePath, object : ThumbnailSpriteCallback {
            override fun onThumbnailSpriteGenerated(spritePath: String) {
                // We can do stuff after thumbnail sprite is generated
            }

            override fun onThumbnailSpriteGenerationError(error: Exception) {
                // We can do stuff when thumbnail sprite generation fails, i.e. log to logcat
                Log.e("error", error.toString())
            }
        })

                // Use asynchronous so it doesn't stop loading and does this as a background task
        AsyncTask.execute {
            spriteGenerator.generateThumbnailSprite()
        }
    }
}

[Luna712]

@LagradOst any thoughts on the above, is it worth me doing the work to learn the codebase here to attempt implementing it and finishing the FFmpeg work, or is it not worth it, and something you don't feel you'd merge if I do?

[LagradOst]

@LagradOst any thoughts on the above, is it worth me doing the work to learn the codebase here to attempt implementing it and finishing the FFmpeg work, or is it not worth it, and something you don't feel you'd merge if I do?

we accept all prs, however ffmpeg will be quite hard to implement, so I suggest just doing the thumbnail only for downloaded vids using what you found

[LagradOst]

dont worry about saving it to a file, you can store the thumbnails in memory (as scoped storage is VERY VERY fucked)

[Luna712]

@LagradOst My new version ended up getting FFmpeg working to basic levels (only supports certain formats right now) and only stores via memory not to a file (except for ffmpeg, it uses a temporary cache file in internal storage). The basic online files I have tested on so far, it does work. So basic online (FFmpeg) support as well as local video support in my new version, but I still have to learn how the code base for CloudStream works to attempt any sort of PR. I also have to figure out how to do the seekbar layout.

[Luna712]

@LagradOst To be honst this was the easy part for me, learning how the UI part works is actually the complex part for me because I haven't done Kotlin before, only able to learn it fast because I have done other languages before so have enough knowledge to figure this out, but I am not sure I'll be able to do the UI.

Now, I managed to get the FFmpeg part tested and working for an mp4, that is 2hrs, 2GB long (from SuperStream), and it takes a while to generate, even though I managed to significantly improve performance, but it does work. Local video thumbnail generation with MediaMetadataRetriever, fully generates almost instantly. I made almost every improvement I wanted to after the above code. It now:

• Just uses rows and columns variables to set interval, it will calculate interval based on video total duration and columns and rows, and will fill up the grid, but includes frames from start to finish of the entire video in proper intervals. This also means it works for videos shorter than any defined interval as well.
• Calculates the width and height based on the video dimensions, then takes the initial frame, calculates dimensions and ensures aspect ratio is maintained to prevent it from loosing aspect ratio in frames on finalized sprite.

Now, to be honest, I am not sure I will be able to do the UI implementation myself, at least probably not any time soon. I can share the finished test code I have if you or anyone else wish to use it in UI, or give me some tips on where I'd implement it, because like I mentioned above ironically, this is the easy part for me, and I haven't done much with Kotlin before. I just //really// want this feature in CloudStream to make it so much more convenient to play videos in CloudStream, as while offline videos is a bit easier to seek, unless you know your exact position in an online video, it is very hard to seek and find a specific part due to buffering, etc... so this feature would make this so much more convenient.

I do think enabling the preview seekbar should be a setting (disabled by default (maybe?)) though, mostly because generating a thumbnail sprite from an online video could cause a lot more data usage in pulling the video frames from online sources. At least at first it should be a setting, whether it always will be I guess is up to you, and might be able to be further improved in the future it doesn't need to be, or at least something enabled by default.

[LagradOst]

@Luna712 create a draft PR so I can take a look at the code

[Luna712]

@Luna712 create a draft PR so I can take a look at the code

Okay, so I don't know how to implement it to CloudStream but I got a sprite generator code going, online (full length, and fairly large) videos work within 30 seconds it generates the sprite now. I feel like an idiot though because MediaMetadataRetriever works for online files as well. So I used that which isna lot faster, whilst keeping FFmpeg/FFprobe for use with supporting m3u8 files, it can be determined if it is even worth keeping, but I can just paste you the code I did create here if you want?

[LagradOst]

@Luna712 create a draft PR so I can take a look at the code

Okay, so I don't know how to implement it to CloudStream but I got a sprite generator code going, online (full length, and fairly large) videos work within 30 seconds it generates the sprite now. I feel like an idiot though because MediaMetadataRetriever works for online files as well. So I used that which isna lot faster, whilst keeping FFmpeg/FFprobe for use with supporting m3u8 files, it can be determined if it is even worth keeping, but I can just paste you the code I did create here if you want?

I dont really want a huge dependency like ffmpeg when android has built in support. Just create a pr w the mediameta

[Luna712]

@Luna712 create a draft PR so I can take a look at the code

Okay, so I don't know how to implement it to CloudStream but I got a sprite generator code going, online (full length, and fairly large) videos work within 30 seconds it generates the sprite now. I feel like an idiot though because MediaMetadataRetriever works for online files as well. So I used that which isna lot faster, whilst keeping FFmpeg/FFprobe for use with supporting m3u8 files, it can be determined if it is even worth keeping, but I can just paste you the code I did create here if you want?

I dont really want a huge dependency like ffmpeg when android has built in support. Just create a pr w the mediameta

Done, #637

[Luna712]

@LagradOst You're awesome! Thank you so very much for finishing this!