Some delays observing while screen share android

[vmeditab]

I am feeling some delays while sharing screen from tablet to android mobile.

Below is my code,

`class CustomVideoCapturer : BaseVideoCapturer() {
 companion object {
    const val TAG = "CustomVideoCapturer"

    const val pixelFormat = ABGR;
 }

private var width: Int = 640
private var height: Int = 480
private var isCapturing: Boolean = false

override fun init() {
    Log.e(TAG, "On Init")
}

override fun onResume() {
    Log.e(TAG, "On Resume")
}

override fun onPause() {
    Log.e(TAG, "On Pause")
}

override fun getCaptureSettings(): CaptureSettings {
    val captureSettings = CaptureSettings()
    captureSettings.width = width
    captureSettings.height = height
    captureSettings.fps = 7
    captureSettings.expectedDelay = 0

    return captureSettings
}

override fun startCapture(): Int {
    Log.e(TAG, "Start Capture")
    return 0
}

override fun stopCapture(): Int {
    Log.e(TAG, "Stop Capture")
    return 0
}

override fun isCaptureStarted(): Boolean {
    return isCapturing
}

override fun destroy() {
    Log.e(TAG, "Destroy")
}

fun sendFrame(imageBuffer: ByteBuffer, imageWidth: Int, imageHeight: Int) {
    provideBufferFrame(imageBuffer, pixelFormat, imageWidth, imageHeight, 0, false)
}

}`

And below is screen sharing code,

        ` screenPublisher =
            Publisher.Builder(this@NewTelevisitActivity).name(jsonObject.toString())
                .capturer(customVideoCapturer).build()
        screenPublisher?.publisherVideoType =
            PublisherKit.PublisherKitVideoType.PublisherKitVideoTypeScreen
        screenPublisher?.publishVideo = true
        screenPublisher?.publishAudio = publisher?.stream?.hasAudio() == true
        screenPublisher?.audioFallbackEnabled = false
        screenPublisher?.setPublisherListener(this)
        session?.publish(screenPublisher)`

As you can see in attached video, while I am expanding/collapsing framelayout, the screen shared video will be shown in black and if I will do any action in tab the screen shared video will be shown again or the frames will be again visible withing 10 to 12 seconds. Please suggest me if any kind of resolution issue or something else.

https://github.com/opentok/opentok-android-sdk-samples/assets/79642753/c3e14db2-f932-484c-afe2-a1e358f09772

[vmeditab]

Hello @v-kpheng, is there any update regarding this issue?

[goncalocostamendes]

@vmeditab I have noted we were not providing a Kotlin sample for screen sharing using a WebViewer so I have created one. At the moment it was not merged yet, but you can have a look at it in https://github.com/opentok/opentok-android-sdk-samples/pull/486.

In regards to your specific problem, I see you have setted the video type of the published stream to PublisherKitVideoType.PublisherKitVideoTypeScreen, which optimizes the video encoding for screen sharing. You have also setted a low frame rate, which is advised, however it seems you have not provided the entire code of your CustomVideoCapturer class:

• With each periodicity is sendFrame() called?
• When do you start sharing the screen? I see startCapture() is not doing anything.

Also your issue is a bit more complex than only sharing the screen, since you are sharing the screen from your tablet to your Android mobile, which is where you expand/collapse the frameLayout. It is possible when these events are triggered, the code is not being handled correctly.

Could you share more code of your project?

[vmeditab]

Hey @goncalocostamendes ,

Please see my code below for activity where I am handling expand collapse framelayout. I am using recyclerview for below layout of the main view and simply used notifyItemchanged methods while pin/unpin.

MainActivity.kt

//This method while sending the frame
    override fun sendFrame(imageBuffer: ByteBuffer, width: Int, height: Int) {
        customVideoCapturer?.sendFrame(imageBuffer, width, height)
    }

// This is used for swapping framelayouts
      attendeesAdapter =
            AttendisAdapter(this@NewTelevisitActivity, object : AddViewDetailClickListener {
                override fun onSwapViews(
                    container: NewSubscriberContainer?, index: Int
                ) {
                    binding.fmSubscriber.restore()
                    subscriberContainer?.let { mSubscribers.set(index, it) }
                    attendeesAdapter.notifyItemChanged(index)
                    subscriberContainer = container

                    updateMainView()
                }
            })

//This is used for update the main large view which is showing in upper side

    private fun updateMainView() {
        binding.fmSubscriber.removeAllViews()
        subscriberContainer?.subscriber?.let {
            it.view.removeSelf()
            (it.view as GLSurfaceView).setZOrderOnTop(false)

            // set ratio for display for web screen share
            if (subscriberContainer?.userDetails?.platform == "web" && subscriberContainer?.userDetails?.designation_desc == "Screen") {
                // PP-608 (Reopen Point - 1)
                // added zoom in zoom out
                val layoutParams = FrameLayout.LayoutParams(
                    binding.fmSubscriber.width,
                    (binding.fmSubscriber.height / 3.5).toInt()
                )
                it.view.layoutParams = layoutParams
                layoutParams.gravity = Gravity.CENTER
                binding.fmSubscriber.addView(it.view)

                it.setStyle(
                    BaseVideoRenderer.STYLE_VIDEO_SCALE,
                    BaseVideoRenderer.STYLE_VIDEO_FIT
                )
            } else {
                binding.fmSubscriber.addView(it.view)
                it.setStyle(
                    BaseVideoRenderer.STYLE_VIDEO_SCALE,
                    if (subscriberContainer?.userDetails?.platform == "tab") BaseVideoRenderer.STYLE_VIDEO_FILL else BaseVideoRenderer.STYLE_VIDEO_FIT
                )
            }
            binding.fmSubscriber.enableTouch(subscriberContainer?.userDetails?.designation_desc == "Screen")
            binding.rlVideo.visibility = if (it.stream?.hasVideo() == true) GONE else VISIBLE
            setVideoListener(it, binding, subscriberContainer)
            setStreamListener(it, binding)
            binding.imgPinnedMute.visibility = VISIBLE
            // PP-745
            // update mic icon of subscriber for main view
            binding.imgPinnedMute.setImageResource(if (it.stream?.hasAudio() == true) R.drawable.ic_pin_mic else R.drawable.ic_pin_mute)
        }
        subscriberContainer?.publisher?.let {
            it.view.removeSelf()
            // PP-608 (Reopen Point - 1)
            // added zoom in zoom out
            binding.fmSubscriber.addView(it.view)
            it.setStyle(
                BaseVideoRenderer.STYLE_VIDEO_SCALE,
                BaseVideoRenderer.STYLE_VIDEO_FILL
            )
            binding.rlVideo.visibility = if (it.stream?.hasVideo() == true) GONE else VISIBLE
            binding.imgPinnedMute.visibility = GONE
            binding.fmSubscriber.enableTouch(false)
        }
        binding.tvSubscriber.text = subscriberContainer?.name
        binding.tvShortName.text =
            ConstTM.getShortName(subscriberContainer?.userDetails?.role_name)
    }

Adapter class

class AttendisAdapter(
    private val context: Context,
    private val addViewDetailClickListener: AddViewDetailClickListener
) :
    ListAdapter<NewSubscriberContainer, AttendisAdapter.AttendeeViewHolder>(
        AttendisDiffCallback()
    ) {

    companion object {
        var intUpdateVideo = -1
        var isPublishVideo = false
    }

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): AttendeeViewHolder {
        return AttendeeViewHolder.from(parent)
    }

    override fun onBindViewHolder(holder: AttendeeViewHolder, position: Int) {
        holder.bind(getItem(position), addViewDetailClickListener, holder.absoluteAdapterPosition, context)
    }

    fun updateVideo(index: Int, publishVideo: Boolean?) {
        intUpdateVideo = index
        isPublishVideo = publishVideo == true
    }

    override fun getItemId(position: Int): Long = position.toLong()

    class AttendeeViewHolder(
        val binding: ProviderListItemBinding
    ) :
        RecyclerView.ViewHolder(binding.root) {
        fun bind(
            subscriberContainer: NewSubscriberContainer,
            addViewDetailClickListener: AddViewDetailClickListener,
            position: Int,
            context: Context,
        ) {

            binding.fmSubscriber.removeAllViews()

            subscriberContainer.subscriber?.let {
                it.view.removeSelf()
                (it.view as GLSurfaceView).setZOrderMediaOverlay(true)
                if (subscriberContainer.userDetails?.platform == "web"  && subscriberContainer.userDetails?.designation_desc == "Screen") {
                    val layoutParams = LayoutParams(
                        binding.fmSubscriber.width,
                        binding.fmSubscriber.height
                    )
                    it.view.layoutParams = layoutParams
                    layoutParams.gravity = Gravity.CENTER
                    binding.fmSubscriber.addView(it.view)
                } else {
                    binding.fmSubscriber.addView(it.view)
                    it.setStyle(
                        BaseVideoRenderer.STYLE_VIDEO_FIT,
                        BaseVideoRenderer.STYLE_VIDEO_FIT
                    )
                }
                binding.imgMuteAudio.setImageResource(ConstTM.setAudioIcon(it.stream?.hasAudio() == true))
                binding.rlVideo.visibility =
                    if (it.stream?.hasVideo() == true) GONE else VISIBLE
                binding.imgMuteAudio.visibility = VISIBLE
                ConstTM.setVideoListener(it, binding, subscriberContainer)
                ConstTM.setStreamListener(it, binding)
            }
            subscriberContainer.publisher?.let {
                it.view.removeSelf()
                (it.view as GLSurfaceView).setZOrderMediaOverlay(true)
                binding.fmSubscriber.addView(it.view)
                it.setStyle(
                    BaseVideoRenderer.STYLE_VIDEO_FIT,
                    BaseVideoRenderer.STYLE_VIDEO_FIT
                )
                binding.imgMuteAudio.setImageResource(ConstTM.setAudioIcon(it.stream?.hasAudio() == true))
                binding.imgMuteAudio.visibility = GONE
                binding.rlVideo.visibility = if (it.publishVideo) GONE else VISIBLE
            }
            binding.tvSubscriber.text = subscriberContainer.name

            if (intUpdateVideo == position) {
                intUpdateVideo = -1
                binding.imgMuteAudio.visibility = GONE
                binding.rlVideo.visibility =
                    if (isPublishVideo) GONE else VISIBLE
                binding.tvSubscriber.text = context.getString(R.string.you)
            }

            binding.tvShortName.text =
                ConstTM.getShortName(subscriberContainer.userDetails?.role_name)

            binding.imgPin.setOnClickListener {
                addViewDetailClickListener.onSwapViews(
                    subscriberContainer,
                    position
                )
            }
        }

        companion object {
            fun from(parent: ViewGroup): AttendeeViewHolder {
                val layoutInflater = LayoutInflater.from(parent.context)
                val binding = ProviderListItemBinding.inflate(layoutInflater, parent, false)
                return AttendeeViewHolder(binding)
            }
        }
    }
}

interface AddViewDetailClickListener {
    fun onSwapViews(
        container: NewSubscriberContainer?,
        index: Int
    )
}

class AttendisDiffCallback : DiffUtil.ItemCallback<NewSubscriberContainer>() {
    override fun areItemsTheSame(
        oldItem: NewSubscriberContainer,
        newItem: NewSubscriberContainer
    ): Boolean = oldItem.providerId == newItem.providerId

    override fun areContentsTheSame(
        oldItem: NewSubscriberContainer,
        newItem: NewSubscriberContainer
    ): Boolean = oldItem.providerId == newItem.providerId
}

MediaProjection.kt

class MediaProjectionService : Service(), ImageReader.OnImageAvailableListener {
    companion object {
        const val TAG = "MediaProjectionService"

        const val FOREGROUND_SERVICE_ID = 1234

        const val NOTIFICATION_CHANNEL_ID = "MediaProjectionService"
        const val NOTIFICATION_CHANNEL_NAME = "Vonage Video"

        const val SCREEN_CAPTURE_NAME = "screencapture"
        const val MAX_SCREEN_AXIS = 1024
        const val VIRTUAL_DISPLAY_FLAGS =
            DisplayManager.VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY or
                    DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC
    }

    private var binder: MediaProjectionBinder? = null
    private var binderIntent: Intent? = null

    private var width = 240
    private var height = 320
    private var density = 0

    private var imageReader: ImageReader? = null
    private var virtualDisplay: VirtualDisplay? = null

    private lateinit var mediaProjection: MediaProjection

    override fun onBind(intent: Intent?): IBinder? {
        Log.d(TAG, "On Bind")
        binder = MediaProjectionBinder()
        binderIntent = intent

        val notification = createNotification()
        startForeground(FOREGROUND_SERVICE_ID, notification, ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION)

        // Get Media Projection
        Log.d(TAG, "Getting Media Projection")
        val resultCode = intent?.getIntExtra("resultCode", Activity.RESULT_CANCELED) ?: 0
        val data = intent?.getParcelableExtra("data") ?: Intent()
        val projectionManager =
            getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
        mediaProjection = projectionManager.getMediaProjection(resultCode, data)

        // Get Display
        Log.d(TAG, "Getting Display")
        val windowManager = getSystemService(Context.WINDOW_SERVICE) as WindowManager
        val display = windowManager.defaultDisplay

        // Metrics
        Log.d(TAG, "Getting Metrics")
        val metrics = DisplayMetrics()
        display.getMetrics(metrics)

        // Size
        Log.d(TAG, "Getting Size")
        val size = Point()
        display.getRealSize(size)
        Log.d(TAG, "Size: ${size.x} x ${size.y}")
        width = size.x
        height = size.y
//        resizeDisplaySizes(size.x, size.y)

        // Density
        Log.d(TAG, "Getting Density")
        density = metrics.densityDpi
        Log.d(TAG, "Density: $density")

        // Create Virtual Display
        createVirtualDisplay()

        return binder
    }

    override fun onUnbind(intent: Intent?): Boolean {
        Log.d(TAG, "On Unbind")
        binderIntent = null
        binder = null

        this.stopForeground(STOP_FOREGROUND_DETACH)
        this.stopSelf()
        return super.onUnbind(intent)
    }

    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        Log.d(TAG, "On Start Command")

        // Do not allow system to restart the service (must be started by user)
        return START_NOT_STICKY
    }

    private fun createVirtualDisplay() {
        Log.i(TAG, "Creating Virtual Display [$width x $height]")

        // Create Image Reader
        imageReader = ImageReader.newInstance(width, height, PixelFormat.RGBA_8888, 2)

        mediaProjection.registerCallback(object : MediaProjection.Callback() {
            override fun onStop() {
                virtualDisplay?.release()
                mediaProjection.unregisterCallback(this)
            }
        }, null)

        // Create Virtual Display
        virtualDisplay = mediaProjection.createVirtualDisplay(
            SCREEN_CAPTURE_NAME,
            width,
            height,
            density,
            VIRTUAL_DISPLAY_FLAGS,
            imageReader?.surface,
            null,
            null
        )

        // Setup Image Available Listener
        try {
            Log.d(TAG, "Setting Image Available Listener")
            val handler = Handler(Looper.getMainLooper())
            imageReader?.setOnImageAvailableListener(this, handler)
        } catch (ex: Exception) {
            Log.e(TAG, ex.message, ex)
        }
    }

    override fun onImageAvailable(reader: ImageReader?) {
        val image = reader?.acquireLatestImage() ?: return
        image.use {
            // Get Image Buffer
            val imagePlane = image.planes[0]
            val imageBuffer = imagePlane.buffer

            // Compute Width (to avoid image distortion on certain devices)
            val rowStride = imagePlane.rowStride
            val pixelStride = imagePlane.pixelStride
            val width = rowStride / pixelStride

            // Send Image Frame Data
            sendFrame(imageBuffer, width, image.height)
        }
    }

    private fun sendFrame(imageBuffer: ByteBuffer, width: Int, height: Int) {
        binder?.mediaProjectionHandler?.sendFrame(imageBuffer, width, height)
    }

    private fun createNotificationChannel() {
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            val channel = NotificationChannel(
                NOTIFICATION_CHANNEL_ID,
                NOTIFICATION_CHANNEL_NAME,
                NotificationManager.IMPORTANCE_DEFAULT
            )
            channel.description = "Vonage Video Media Projection Service"

            val notificationManager =
                getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager
            notificationManager.createNotificationChannel(channel)
        }
    }

    private fun createNotification(): Notification {
        createNotificationChannel()

        return if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            Notification.Builder(this, NOTIFICATION_CHANNEL_ID)
                .setSmallIcon(R.drawable.launch_background)
                .setContentTitle("Title")
                .setContentText("Running screenshare video driver")
                .build()
        } else {
            Notification.Builder(this)
                .setSmallIcon(R.drawable.launch_background)
                .setContentTitle("Title")
                .setContentText("Running screenshare video driver")
                .setPriority(Notification.PRIORITY_DEFAULT)
                .build()
        }
    }
}

I am using above service to create virtual display with frame.

[v-kpheng]

@vmeditab, a new sample app was created: https://github.com/opentok/opentok-android-sdk-samples/tree/main/Screen-Sharing-Kotlin. Can you still reproduce with that?