mobile_scanner overlay on iOS webview

[AnikethFitPass]

Description: I'm experiencing an issue where the mobile_scanner widget (version 3.4.1) displays an unexpected overlay when used within an iOS webview integrated into a native iOS app. This overlay contains controls such as pause/play, mute, a cross button, and a "Live" label at the bottom. The website works correctly on Safari, Chrome, and Android webviews, suggesting the issue is specific to iOS webviews.

Steps to Reproduce: Integrate a website containing a mobile_scanner widget into an iOS native app using a webview. Run the iOS app and attempt to use the mobile_scanner widget.

Expected Behavior: The mobile_scanner widget should function normally within the webview without any overlay.

Actual Behavior: An overlay with multimedia controls appears on top of the mobile_scanner widget, interfering with its functionality. Additional Information: The issue persists even after upgrading to the latest versions of mobile_scanner. The problem only occurs within iOS webviews, not on Safari, Chrome, or Android webviews.

The below shown screen is what I am experiencing

The below shown screen is the expected

[AnikethFitPass]

@juliansteenbakker @navaronbracke Please help

[navaronbracke]

I wonder, are the media controls shown because we do not set extra options on the video element?

We probably need to provide additional configuration as specified in https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video but I'm not sure what we should set yet.

Does adding autoplay playsinline to the video element fix it in such a webview?

[navaronbracke]

So I did a bit of digging. Apparently this bug might exist because, the controls attribute is being set. If you inspect the webview, do you see if the video has the controls attribute set? And does the controls attribute appear again when playing/pausing the video?

We might need to do something like

    videoElement.controls = false;

    videoElement.addEventListener('play', function () {
      this.controls = false;
    });

    videoElement.addEventListener('pause', function () {
      this.controls = false;
    });

[navaronbracke]

Sorry, I mean if you inspect the WKWebView (you'll have to set it to inspectable on iOS 16.4+) and then view the <video> element in the inspector, does it have the controls attribute when playing and/or paused?

I provided similar guidance on using the inspector in https://github.com/juliansteenbakker/mobile_scanner/issues/950#issuecomment-2136595869

[navaronbracke]

I'm going to try and see if I can reproduce this locally with a standalone iOS app that loads mobile scanner in a WKWebview, using the locally built web version as a bundle resource for the webview and load it using the MainBundle

[AnikethFitPass]

i have not added any specific attributes to the ios webview

`import UIKit import WebKit

class ViewController: UIViewController, UITextFieldDelegate, WKNavigationDelegate, WKUIDelegate {

let webView = WKWebView()
let urlTextField = UITextField()
let loadButton = UIButton()
let floatingButton = UIButton()

override func viewDidLoad() {
    super.viewDidLoad()
    setupViews()
}

func setupViews() {
    // Set up WebViewConfiguration
    let preferences = WKPreferences()
    preferences.javaScriptEnabled = true

    let configuration = WKWebViewConfiguration()
    configuration.preferences = preferences

    // Enable DOM Storage
    configuration.websiteDataStore = .default()

    webView.navigationDelegate = self
    webView.uiDelegate = self

    // Set up WebView
    webView.frame = view.bounds
    webView.autoresizingMask = [.flexibleWidth, .flexibleHeight]
    view.addSubview(webView)

    // Set up URL TextField
    urlTextField.frame = CGRect(x: 20, y: -40, width: view.bounds.width - 120, height: 40)
    urlTextField.borderStyle = .roundedRect
    urlTextField.placeholder = "Enter URL"
    urlTextField.delegate = self
    urlTextField.autoresizingMask = [.flexibleWidth, .flexibleTopMargin]
    view.addSubview(urlTextField)

    // Set up Load Button
    loadButton.frame = CGRect(x: view.bounds.width - 100, y: view.bounds.height - 60, width: 80, height: 40)
    loadButton.setTitle("Load", for: .normal)
    loadButton.setTitleColor(.blue, for: .normal)
    loadButton.addTarget(self, action: #selector(loadButtonTapped), for: .touchUpInside)
    loadButton.autoresizingMask = [.flexibleTopMargin, .flexibleLeftMargin]
    view.addSubview(loadButton)

    // Set up Floating Button
    floatingButton.frame = CGRect(x: view.bounds.width - 70, y: view.bounds.height - 130, width: 60, height: 60)
    floatingButton.backgroundColor = .systemBlue
    floatingButton.setTitle("+", for: .normal)
    floatingButton.titleLabel?.font = UIFont.systemFont(ofSize: 30)
    floatingButton.layer.cornerRadius = 30
    floatingButton.addTarget(self, action: #selector(floatingButtonTapped), for: .touchUpInside)
    floatingButton.autoresizingMask = [.flexibleTopMargin, .flexibleLeftMargin]
    view.addSubview(floatingButton)

    // Load initial URL
    if let initialURL = URL(string: "https://flutter.dev") {
        let request = URLRequest(url: initialURL)
        webView.load(request)
    }
}

@objc func loadButtonTapped() {
    guard let urlString = urlTextField.text, !urlString.isEmpty, let url = URL(string: urlString) else {
        // Show alert if the URL is invalid or empty
        let alert = UIAlertController(title: "Invalid URL", message: "Please enter a valid URL.", preferredStyle: .alert)
        alert.addAction(UIAlertAction(title: "OK", style: .default))
        present(alert, animated: true)
        return
    }
    let request = URLRequest(url: url)
    webView.load(request)
    urlTextField.resignFirstResponder()  // Dismiss the keyboard
}

@objc func floatingButtonTapped() {
    UIView.animate(withDuration: 0.3) {
        self.urlTextField.frame.origin.y = self.urlTextField.frame.origin.y == 0 ? -40 : 0
    }
}

// Handle keyboard return key press
func textFieldShouldReturn(_ textField: UITextField) -> Bool {
    loadButtonTapped()
    return true
}

// Handle JavaScript dialogs and permissions
func webView(_ webView: WKWebView, runJavaScriptAlertPanelWithMessage message: String, initiatedByFrame frame: WKFrameInfo, completionHandler: @escaping () -> Void) {
    let alert = UIAlertController(title: nil, message: message, preferredStyle: .alert)
    alert.addAction(UIAlertAction(title: "OK", style: .default, handler: { _ in
        completionHandler()
    }))
    present(alert, animated: true)
}

func webView(_ webView: WKWebView, runJavaScriptConfirmPanelWithMessage message: String, initiatedByFrame frame: WKFrameInfo, completionHandler: @escaping (Bool) -> Void) {
    let alert = UIAlertController(title: nil, message: message, preferredStyle: .alert)
    alert.addAction(UIAlertAction(title: "Cancel", style: .cancel, handler: { _ in
        completionHandler(false)
    }))
    alert.addAction(UIAlertAction(title: "OK", style: .default, handler: { _ in
        completionHandler(true)
    }))
    present(alert, animated: true)
}

func webView(_ webView: WKWebView, runJavaScriptTextInputPanelWithPrompt prompt: String, defaultText: String?, initiatedByFrame frame: WKFrameInfo, completionHandler: @escaping (String?) -> Void) {
    let alert = UIAlertController(title: nil, message: prompt, preferredStyle: .alert)
    alert.addTextField { textField in
        textField.text = defaultText
    }
    alert.addAction(UIAlertAction(title: "Cancel", style: .cancel, handler: { _ in
        completionHandler(nil)
    }))
    alert.addAction(UIAlertAction(title: "OK", style: .default, handler: { _ in
        let textField = alert.textFields?.first
        completionHandler(textField?.text)
    }))
    present(alert, animated: true)
}

}

`

this is my swift code created for testing purposes

[navaronbracke]

Your actual Swift / HTML code is irrelevant. Have you inspected the HTML that is loaded in the WKWebview, using the steps from https://github.com/juliansteenbakker/mobile_scanner/issues/950#issuecomment-2136595869 ?

I'll need to see how the video element is loaded in the webview, maybe the webview adds some extra attributes to the video tag, that we should turn off.

[AnikethFitPass]

[navaronbracke]

Aha, we have the controls attribute on the video element. We should probably set that to false explicitly. You can expect a fix shortly.

[AnikethFitPass]

Great, will i have to update it to the newer version of mobile_scanner or can i fix it in the version which I am using ?

[navaronbracke]

If you are already using version 5.x, you will have to migrate to 5.2.3. I don't think we have a policy of backporting fixes to earlier releases.

[AnikethFitPass]

I am using the version 3.4.1, Can i inject the script into the webview via wkwebview ? 🥲

[navaronbracke]

You might be able to use a CSS rule to hide the controls?

[AnikethFitPass]

@navaronbracke I upgraded the mobile_scanner plugin to 5.2.3 but still seem to face the issue .

[navaronbracke]

Hmm, is that media controls container added by the webview? The controls property isn't there anymore

[AnikethFitPass]

https://github.com/react-native-webview/react-native-webview/issues/1672#issuecomment-776531933

[AnikethFitPass]

The controls seem to get removed once a qr code is scanned , but reappears again after restarting the app

[navaronbracke]

The properties https://developer.apple.com/documentation/webkit/wkwebviewconfiguration/1614793-allowsinlinemediaplayback and https://developer.apple.com/documentation/webkit/wkwebviewconfiguration/1851524-mediatypesrequiringuseractionfor - which you linked from that comment in that React Native issue - will need to be set in the webview configuration. We can only provide documentation for that, as we do not control webviews in this plugin.

For example webview_flutter_wkwebview provides https://pub.dev/documentation/webview_flutter_wkwebview/latest/webview_flutter_wkwebview/WebKitWebViewControllerCreationParams-class.html to do that.

[navaronbracke]

The controls seem to get removed once a qr code is scanned , but reappears again after restarting the app

Do you mean that the controls attribute reappears, or is the <div class="media-controls-container"> coming back?

[AnikethFitPass]

I added both the inlinemediaplayback = true and mediaTypesRequiringUserActionForPlayback = [] on my ios webview , still nothing .

[AnikethFitPass]

The controls seem to get removed once a qr code is scanned , but reappears again after restarting the app

Do you mean that the controls attribute reappears, or is the <div class="media-controls-container"> coming back?

Yes

[navaronbracke]

Sorry, is it

1) controls attribute reappears on the video element

or

2) controls attribute is gone from the video element, but the <div class="media-controls-container"> is still there

after an app restart

[AnikethFitPass]

@navaronbracke I upgraded the mobile_scanner plugin to 5.2.3 but still seem to face the issue .

This same element reappears

[AnikethFitPass]

Running the app on Simulator gives this screen when opening the scanner

[navaronbracke]

@AnikethFitPass The iOS Simulator has no camera. Likewise, the Android emulator has a "fake" camera preview (it provides a fixed 3D scene in the viewfinder). So running on a real device is the only "supported" method.