Krosxx / Android-Auto-Api

安卓无障碍服务Api, 为了简化无障碍服务使用,并使用 Kotlin 以提供简洁的Api。
Apache License 2.0
494 stars 83 forks source link

为什么ScreenTextFinder().find()获取到的不是当前界面的,而是旧的界面的元素 #20

Open johnvi-l opened 1 year ago

johnvi-l commented 1 year ago

抓取的内容是在微信webview内部, 设置了每隔3秒钟抓取一次,但是发现webview内容改变了,但是抓取到的还是旧界面内容

Wings-Looo commented 1 year ago

同遇到更新NodeTree更新不及时的问题

Krosxx commented 1 year ago

尝试使用下面 accessibility_config.xml

<?xml version="1.0" encoding="utf-8"?>
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeAllMask"
    android:accessibilityFeedbackType="feedbackAllMask"
    android:accessibilityFlags="flagIncludeNotImportantViews|flagReportViewIds|flagRetrieveInteractiveWindows|flagRequestEnhancedWebAccessibility"
    android:canPerformGestures="true"
    android:canRequestEnhancedWebAccessibility="true"
    android:canRetrieveWindowContent="true"
    android:canTakeScreenshot="true"
    android:summary="" />

再尝试,targetSdk 设为 31

johnvi-l commented 1 year ago

尝试使用下面 accessibility_config.xml

<?xml version="1.0" encoding="utf-8"?>
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeAllMask"
    android:accessibilityFeedbackType="feedbackAllMask"
    android:accessibilityFlags="flagIncludeNotImportantViews|flagReportViewIds|flagRetrieveInteractiveWindows|flagRequestEnhancedWebAccessibility"
    android:canPerformGestures="true"
    android:canRequestEnhancedWebAccessibility="true"
    android:canRetrieveWindowContent="true"
    android:canTakeScreenshot="true"
    android:summary="" />

再尝试,targetSdk 设为 31

尝试过了 还是不行

Krosxx commented 1 year ago

试试主动清除缓存: https://github.com/google/android-uiconductor/blob/master/xmldumper/app/src/main/java/com/google/uicd/xmldumper/utils/AndroidPlatformReflectionUtils.java#L33C1-L48C4

byteboycn commented 8 months ago

试试主动清除缓存: https://github.com/google/android-uiconductor/blob/master/xmldumper/app/src/main/java/com/google/uicd/xmldumper/utils/AndroidPlatformReflectionUtils.java#L33C1-L48C4

有用

ysy950803 commented 5 months ago

在原生回调 onAccessibilityEvent 中去find内容,会比较好用,作者自己搞的这个onPageUpdate条件较为严格。给个参考:

    private var findJob: Job? = null

    // 启用页面更新回调
    override val enableListenPageUpdate: Boolean = true

    // 页面更新回调
    override fun onPageUpdate(currentScope: AppScope) {
        logForDebug("onPageUpdate $currentScope")
    }

    override fun onAccessibilityEvent(event: AccessibilityEvent?) {
        super.onAccessibilityEvent(event)
        logForDebug("onAccessibilityEvent $event")
        findJob?.cancel()
        findJob = launchGlobal {
            delay(1000L)
            runCatching {
                val screenText = ScreenTextFinder().find().joinToString("\n")
                logForDebug("onAccessibilityEvent $screenText")
            }.onFailure {
                logForDebug("$it")
            }
        }
    }

原因分析,从作者的源码中能看到updateCurrentApp有很多if条件过滤,这种过滤对于支付宝微信那种Flutter或者RN渲染原生控件的就回调不到onPageUpdate,因为Activity并不会发生切换,但实际上内容已经变了:

    override fun onAccessibilityEvent(event: AccessibilityEvent) {
        if (!enableListenPageUpdate) return
        if (event.eventType == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED) {
            // 界面切换
            val classNameStr = event.className
            val pkg = event.packageName as String?
            if (!classNameStr.isNullOrBlank() && pkg != null) {
                GlobalScope.launch {
                    updateCurrentApp(pkg, classNameStr.toString())
                }
            }
        }
    }

    /**
     * 更新当前[currentScope]
     * @param pkg String
     * @param pageName String Activity or Dialog
     */
    private fun updateCurrentApp(pkg: String, pageName: String) {
        if (currentScope?.packageName == pkg && pageName == currentScope?.pageName) {
            return
        }
        if (
            pageName.startsWith("android.widget") ||
            pageName.startsWith("android.view") ||
            pageName.startsWith("android.inputmethodservice") ||
            pageIsView(pageName)
        ) {
            return
        }

        currentScope = currentScope?.also {
            it.pageName = pageName
            it.packageName = pkg
        } ?: AppScope(pkg, pageName)
        onPageUpdate(currentScope!!)
    }
zkywalker commented 2 months ago
AccessibilityEvent

你说的总体是对的,但是翻阅下源码,root是从accessibilityService里取的windows,而如果你打印onAccessibilityEvent的时候会发现很多情况下Webview并不会触发onAccessibilityEvent,而不是因为源码过滤掉了。这个问题我给出的解决方案是故意触发一次onAccessibilityEvent,例如制造一个弹窗事件然后隐藏,之后再去读取windows就是正确的了

shunlibest commented 5 days ago

试试主动清除缓存: https://github.com/google/android-uiconductor/blob/master/xmldumper/app/src/main/java/com/google/uicd/xmldumper/utils/AndroidPlatformReflectionUtils.java#L33C1-L48C4

这个方法好像在高版本废弃了,在Android31 之后,使用AccessibilityService.clearCache() 方法,试了一下,好像有用。

Krosxx commented 5 days ago

@shunlibest 是的,clearCache 参数变了

public static boolean clearAccessibilityCache() {
        boolean success = false;

        try {
            final Class<?> c = Class.forName("android.view.accessibility.AccessibilityInteractionClient");
            final Method getInstance = AndroidPlatformReflectionUtils.method(c, "getInstance");
            final Object instance = getInstance.invoke(null);
            try {
                final Method clearCache = method(instance.getClass(), "clearCache");
                clearCache.invoke(instance);
            } catch (Throwable e) {
                SparseArray<?> cache = (SparseArray<?>) getField(c, "sConnectionCache", null);
                final Method clearCache = method(instance.getClass(), "clearCache", int.class);
                for (int i = 0; i < cache.size(); i++) {
                    int conId = cache.keyAt(i);
                    clearCache.invoke(instance, conId);
                    Timber.i("clear cache success with %s", conId);
                }
            }
            success = true;
        } catch (ReflectiveOperationException | RuntimeException e) {
            RemoSocket.INSTANCE.sendError("Failed to clear Accessibility Node cache. " + e.getMessage());
        }
        return success;
    }

效果不知是否和AccessibilityService.clearCache() 一致

JChunyu commented 3 days ago

这个东西在 WebView 场景下本来就是不使用的