Cannot interact with UI inside of Safari Web View Controller

[JoeGaggler]

Maestro miscalculates the geometry of elements shown inside of Safari Web View Controllers, which prevents interactions such as tapOn.

Here is a screenshot of Maestro Studio showing the incorrect frame for the Username TextField, with my annotations in red to indicate my guess that Maestro is not incorporating Safari's offset from the top of the screen:

[amanjeetsingh150]

Can you share the link to this web page, the app flow, or any example I can use to reproduce this locally? Feel free to send a DM on the public channel here.

[JoeGaggler]

I sent a DM, but for everyone's interest: create a new blank SwiftUI project in Xcode, and then replace ContentView.swift with the following to repro:

import SwiftUI
import SafariServices

struct ContentView: View {
    @State var siteUrl = URL(string: "https://github.com/login")

    var body: some View {
        SafariView(url: $siteUrl)
            .background(Color.blue)
    }
}

struct SafariView: UIViewControllerRepresentable {
    @Binding var url: URL?

    func makeUIViewController(context: Context) -> UIViewController {
        let config = SFSafariViewController.Configuration()

        let actualURL = self.url! // TODO: handle error when URL is nil
        let vc = SFSafariViewController(url: actualURL, configuration: config)
        vc.dismissButtonStyle = .cancel
        return vc
    }

    func updateUIViewController(_ uiViewController: UIViewController, context: Context) {

    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

[JoeGaggler]

I believe this is due to SFSafariViewController using a separate displayID from the main app in order to sandbox the web view.

When I dumped the contents of app.snapshot().dictionaryRepresentation, I found that the web page elements are associated with displayID: 0 instead of 1 as with other UI elements (shown in red boxes):

Because of this, I believe that the frame calculation must be made relative to the parent element on display: 1, which in this case has a Y-offset of 59 (see green arrow).

I was unable to find any Apple documentation regarding displayID, so I am not sure how to confirm.

[axelniklasson]

Thanks for taking the time to raise this issue! There has been a lot of improvements and fixes on Maestro since this was raised so we’ll go ahead and close this one out as part of an issue cleanup - if you still experience issues, please open a new issue with reproduction steps. Thanks again for using maestro!

[JoeGaggler]

I retested using the latest version, and unfortunately this issue is still reproducible:

[axelniklasson]

Thanks for checking @JoeGaggler, seems this is still a problem - reopening the issue