search

AndreyPavlenko / Fermata

GNU General Public License v3.0
526 stars 68 forks source link

Fermata mirror - can see screen projection but cannot register any touch screen input? #455

Open seemebreakthis opened 2 months ago

seemebreakthis commented 2 months ago

Something seems really odd, it is not problematic across the board. The touchscreen does seem to work in these areas:

Lock screen, i.e. I can unlock by entering the correct pattern for the password etc

Lock screen notifications, so I can scroll there etc

Within Fermata auto itself whenever I can somehow get it to run in the foreground on the phone

Google maps (the phone version, not talking about the AA version)

System navigation gestures (swipe from the side to go back, swipe from the top to show notifications, etc)

The 'main menu screen' of Fermata mirror

... But touch screen stops working when the phone shows anything else that I have tried. Phone launcher's home screen, other apps, erc... touch screen just stops responding. What's particularly interesting is, whenever touch screen on the head unit doesn't work, the touchscreen on the phone itself doesn't work either, i.e. I basically lose the ability to interact with the phone, except for the system navigation gestures which COULD make the notifications panel appear etc.

My phone's touchscreen regains control as soon as I disconnect from AA.

Also, don't know if it is relevant, but I have issued the "appops set me.aap.fermata.auto.dear.google.why PROJECT_MEDIA allow" command as root. But I have since uninstalled / reinstalled fermata auto several times (plus reactivating fermata auto in LSposed) as part of my effort to fix the problem.

Please help. Thank you !

AndreyPavlenko commented 2 months ago

This is probably due to the Xposed hook that allows direct communication with other apps without using the accessibility service. Do you have any other apps, except AA, enabled in the LSPosed Fermata module configuration?

seemebreakthis commented 2 months ago

Just the defaults... I didn't change anything from the recommended (and checked by default) check boxes. So, Android Auto, and Maps.

Thanks, will certainly contribute by donating if things work out at the end. Thanks again !

Screenshot_2024-03-29-02-48-02-526-edit_com android shell

seemebreakthis commented 2 months ago

Just tried unchecking Maps, still problem, the only difference being now Google Maps also doesn't respond to touch on either the phone or the head unit...

AndreyPavlenko commented 2 months ago

Seems this is something device specific, because there should not be any impact on the phone touch screen, it should always work. Have you tried to disable the Xposed module and restart the phone?

NeXotics commented 2 months ago

Hi. I have the same problem. Xiaomi redmi note 10 pro. Touch screen only works on main Fermata screen and in Chrome! Any other apps and actions don't work. But once it was amazing and touchscreen worked in other apps, can't repeat that :) Напишу по русски, если можно :) Та же проблема - тач не работает нигде, кроме бразуера хром и рабочего стола Fermata. В xposed галки по умолчанию (в том числе и на хроме). Сегодня утром ехал и каким то чудом тач заработал везде. Но при запуске YouTube (revanced) тач снова отвалился и вернулось все к началу и больше я не могу добиться работы тача...Телефон перезагружал...

NeXotics commented 2 months ago

Seems this is something device specific, because there should not be any impact on the phone touch screen, it should always work. Have you tried to disable the Xposed module and restart the phone?

Hm. Disabling module and reboot was helpful. Will observe some time. Thank you!

seemebreakthis commented 2 months ago

Would like to see if you are interested in looking further into this issue that's hitting some of us, by watching how it manifests itself:

https://github.com/AndreyPavlenko/Fermata/assets/5940900/32e7c40f-6036-4e02-ab56-c0394f56a884

In the video I tried starting Fermata Mirror 4 times. The 1st, 3rd, and 4th time, the touchscreen would work for the first few seconds before it stops working. The 2nd time it wasn't working at all.

And of the numerous other times I tried, about 50% of the time the touchscreen would not respond at all. 50% of the time it would respond for a few seconds before it stops responding. And there are the rare occasions (maybe once every 50 starts) that the touchscreen would stay working for the entire duration of the mirror session (before I exit Fermata Mirror, that is. Then it goes back to the non-working state next time I start Fermata Mirror).

BTW, it is exactly the same behavior and the same issue with my 'official' car head unit. Only once or twice out of a numerous amount of attempts could I use Fermata Mirror (it was really nice when it actually worked).

Coincidentally my phone is also a Xiaomi. Xiaomi 11.

And for me, I cannot disable the LSposed module. Doing that will make the Fermata Auto and Fermata Mirror icons go away on the AA menu.

Thank you.

NeXotics commented 2 months ago

Would like to see if you are interested in looking further into this issue that's hitting some of us, by watching how it manifests itself:

Fermata.mp4 In the video I tried starting Fermata Mirror 4 times. The 1st, 3rd, and 4th time, the touchscreen would work for the first few seconds before it stops working. The 2nd time it wasn't working at all.

And of the numerous other times I tried, about 50% of the time the touchscreen would not respond at all. 50% of the time it would respond for a few seconds before it stops responding. And there are the rare occasions (maybe once every 50 starts) that the touchscreen would stay working for the entire duration of the mirror session (before I exit Fermata Mirror, that is. Then it goes back to the non-working state next time I start Fermata Mirror).

BTW, it is exactly the same behavior and the same issue with my 'official' car head unit. Only once or twice out of a numerous amount of attempts could I use Fermata Mirror (it was really nice when it actually worked).

Coincidentally my phone is also a Xiaomi. Xiaomi 11.

And for me, I cannot disable the LSposed module. Doing that will make the Fermata Auto and Fermata Mirror icons go away on the AA menu.

Thank you.

Hi. Just turn off the the module in lsposed, then reboot your phone and turn on module again. Try that.

seemebreakthis commented 2 months ago

Hi. Just turn off the the module in lsposed, then reboot your phone and turn on module again. Try that.

Just tried. Unfortunately it didn't seem to work. Thank you anyway !

AndreyPavlenko commented 2 months ago

Please, send me the LSPosed module log. Try to disable the module and reboot. Do not enable after reboot.

seemebreakthis commented 2 months ago

Please, send me the LSPosed module log. Try to disable the module and reboot. Do not enable after reboot.

Is this what you are looking for?

modules_2024-04-03T221836.269102.log.zip

(I extracted the log after disabling the fermata module and rebooting. So I think this is why Fermata is nowhere to be found in the log)

AndreyPavlenko commented 2 months ago

Yes, but I need the log when the module is enabled and you have the touch screen issue.

seemebreakthis commented 2 months ago

Yes, but I need the log when the module is enabled and you have the touch screen issue.

Sorry just to make sure I understand you correctly: you would like me to

  1. Disable the fermata module in lsposed
  2. Reboot
  3. Enable the fermata module in lsposed
  4. Reboot again
  5. Capture the lsposed module log after I have started Fermata Mirror and experience the touch screen issue
  6. Send that module log to you

Am I correct?

seemebreakthis commented 2 months ago

Hi @AndreyPavlenko , assuming my procedure is somewhat close to what you need (I cannot run Fermata Mirror anyway if I have the module disabled, so there's probably no other way), this is the log:

modules_2024-04-03T224010.559239.zip

Thank you very much.

AndreyPavlenko commented 2 months ago

I cannot run Fermata Mirror anyway if I have the module disabled

You can with DHU emulator, or use Fermata FS Mirror before disabling the module.

In the log I see that it's hooking into Google Maps. Open the module configuration and uncheck all other apps except AA.

seemebreakthis commented 2 months ago

This time I did this:

  1. Rebooted my phone (with Fermata module enabled in LSposed)
  2. Connected my phone to AA. I can see Fermata Mirror, did not run
  3. Disconnected from AA, then disabled the module
  4. Connected to AA
  5. Ran Fermata Mirror several times (touchscreen not working, exact same symptoms as before, sometimes it worked for a few seconds at first)
  6. Extracted this modules log

Here you go. Anything else you need, please just ask.

modules_2024-04-04T100358.465991.zip

AndreyPavlenko commented 2 months ago

You have not disabled Google Maps. There should not be this message in the log:

Hooking into package com.google.android.apps.maps
seemebreakthis commented 2 months ago

My bad. I was under the impression it didn't matter since I have already disabled the whole module in step 3 above before running Fermata Mirror.

Here you go. Note that the problem persists on my phone even with Google Maps unchecked all the way.

modules_2024-04-04T232337.564936.zip

AndreyPavlenko commented 2 months ago

Hm, then it's something different. You said that the phone touch screen does not work either. Is it true? It should never happen.

seemebreakthis commented 2 months ago

Yeah, whenever the touch screen doesn't work in fermata mirror, the phone touch screen doesn't work either.

But even when touch doesn't work, it will still work specifically for the back and home buttons of fermata mirror. The "swipe from side" or "swipe from top" system-wide gestures would also work, both on the phone and on the head unit.

Maybe the problem is really specific to my phone only.... No one else has reported it am I right?

AndreyPavlenko commented 2 months ago

Right, no one has reported the problem with the phone touch screen. You may try one more trick - start Fermata FS Mirror, make sure it's started properly, disconnect the phone, disable the LSPosed module and restart. Fermata FS Mirror should be started automatically when connected to AA.

seemebreakthis commented 2 months ago

Tried that, didn't work for me either.

One last comment before I give up - touch ALWAYS works in certain areas: 1. when I launch fermata auto in fermata mirror, that farmata auto always responds to touch; 2. The system home, back and menu buttons; 3. Phone's notification screen and control center screen; 4. The main (and only) menu of fermata mirror plus the two buttons rendered by fermata mirror.

It is almost as if fermata mirror doesn't have enough privilege to 'forward' the touch events to other apps, including the launcher app.

AndreyPavlenko commented 2 months ago

Fermata UIs always work, the touch events are transferred to them directly. For the other apps, either the accessibility service or Xposed hook is used. For some reason, the accessibility service does not work properly. You may also try enabling the Xposed hook for some apps. Open the module configuration and check the apps that you want to use in the mirror, restart the phone. Be aware - the hook allows using the selected apps even if the phone is locked.

NeXotics commented 2 months ago

Yeah, whenever the touch screen doesn't work in fermata mirror, the phone touch screen doesn't work either.

But even when touch doesn't work, it will still work specifically for the back and home buttons of fermata mirror. The "swipe from side" or "swipe from top" system-wide gestures would also work, both on the phone and on the head unit.

Maybe the problem is really specific to my phone only.... No one else has reported it am I right?

You are not only one with this problem. I had same too. I'm just turned on apps trigger in module that I use in Fermata Mirror and touchscreen working on it now. But some times it working unstable.

seemebreakthis commented 2 months ago

You are not only one with this problem. I had same too. I'm just turned on apps trigger in module that I use in Fermata Mirror and touchscreen working on it now. But some times it working unstable.

... I thought you got it working by disabling the module altogether?

NeXotics commented 2 months ago

You are not only one with this problem. I had same too. I'm just turned on apps trigger in module that I use in Fermata Mirror and touchscreen working on it now. But some times it working unstable.

... I thought you got it working by disabling the module altogether?

I had not disabled module at all. I had disabled it, reboot and enabled again. And it works. I think that it works normally, but then some times I have same effect unworking touchscreen. Now I enabled triggers in module for apps want to use in Fermata and it work almost normal (sometimes not work on first start).

MichalUngr23 commented 2 months ago

Po aktualizaci mi Fermata vůbec nefunguje telefon mam Xiaomi redmi note 10 android 13 radio octavia 3 Amundsen . Zobrazi se pouze procházení soubory a nic vic

VT53 commented 1 month ago

The same situation can see screen projection but cannot use any touch screen input but I not have root phone(Sony xperia 1 II) , Does Fermata mirror not suppot no-root?

Ircama commented 1 month ago

Thanks for this app. I have to report the same situation. Below the detailed description of my case.

I also did adb shell appops set me.aap.fermata.auto.dear.google.why PROJECT_MEDIA allow

The only mode to activate the screen mirroring is to enable the "Accessibility" setting for Fermata Auto on the smartphone.

When the screen mirroring starts (e.g., run "Fermata Mirror" in AA), also the "Screen cast" toggle for Fermata Auto becomes active.

After activating the screen mirroring, the Android drop-down screen works (swiping down from the top of the screen) and Android settings can be changed. Also swiping up from the bottom to control applications works. Anyway, when starting any application, there is no possibility to interact with it until the screencast is active. Disabling the screencast restores the possibility to interact with the applications, but freezes Fermata Mirror on AA. When clicking again on Fermata Mirror (AA), the Screen cast option is newly enabled and the application becomes unresponsive on the Android device.

Permissions:

Packages:
  Package [me.aap.fermata.auto.dear.google.why] (4f652ac):
    userId=10710
    pkg=Package{719cb75 me.aap.fermata.auto.dear.google.why}
    codePath=/data/app/~~0QNu9S6sH1ypZgikGkHNAA==/me.aap.fermata.auto.dear.google.why-758SNULU4v5uk-_rXpbzWA==
    resourcePath=/data/app/~~0QNu9S6sH1ypZgikGkHNAA==/me.aap.fermata.auto.dear.google.why-758SNULU4v5uk-_rXpbzWA==
    legacyNativeLibraryDir=/data/app/~~0QNu9S6sH1ypZgikGkHNAA==/me.aap.fermata.auto.dear.google.why-758SNULU4v5uk-_rXpbzWA==/lib
    extractNativeLibs=false
    primaryCpuAbi=arm64-v8a
    secondaryCpuAbi=null
    cpuAbiOverride=null
    versionCode=237 minSdk=23 targetSdk=34
    minExtensionVersions=[]
    versionName=1.9.9
    usesNonSdkApi=false
    splits=[base]
    apkSigningVersion=3
    applicationInfo=PackageImpl{719cb75 me.aap.fermata.auto.dear.google.why}
    flags=[ HAS_CODE ALLOW_CLEAR_USER_DATA ALLOW_BACKUP ]
    privateFlags=[ PRIVATE_FLAG_ACTIVITIES_RESIZE_MODE_RESIZEABLE_VIA_SDK_VERSION ALLOW_AUDIO_PLAYBACK_CAPTURE PRIVATE_FLAG_REQUEST_LEGACY_EXTERNAL_STORAGE HAS_DOMAIN_URLS PRIVATE_FLAG_ALLOW_NATIVE_HEAP_POINTER_TAGGING ]
    forceQueryable=false
    queriesIntents=[Intent { act=android.intent.action.MAIN }, Intent { act=android.intent.action.OPEN_DOCUMENT_TREE }, Intent { act=android.intent.action.SENDTO }, Intent { act=android.speech.RecognitionService }, Intent { act=android.intent.action.TTS_SERVICE }, Intent { act=android.speech.tts.engine.INSTALL_TTS_DATA }]
    dataDir=/data/user/0/me.aap.fermata.auto.dear.google.why
    supportsScreens=[small, medium, large, xlarge, resizeable, anyDensity]
    timeStamp=2024-05-03 16:35:25
    firstInstallTime=2024-05-02 21:21:17
    lastUpdateTime=2024-05-03 16:35:26
    installerPackageName=com.google.android.packageinstaller
    signatures=PackageSignatures{377770a version:3, signatures:[d5c7a0ff], past signatures:[]}
    installPermissionsFixed=true
    pkgFlags=[ HAS_CODE ALLOW_CLEAR_USER_DATA ALLOW_BACKUP ]
    declared permissions:
      me.aap.fermata.auto.dear.google.why.DYNAMIC_RECEIVER_NOT_EXPORTED_PERMISSION: prot=signature, INSTALLED
    requested permissions:
      androidx.car.app.ACCESS_SURFACE
      androidx.car.app.NAVIGATION_TEMPLATES
      android.permission.WAKE_LOCK
      android.permission.DISABLE_KEYGUARD
      android.permission.SYSTEM_ALERT_WINDOW
      android.permission.REQUEST_INSTALL_PACKAGES
      android.permission.REQUEST_DELETE_PACKAGES
      android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION
      android.permission.MANAGE_EXTERNAL_STORAGE
      android.permission.INTERNET
      android.permission.ACCESS_NETWORK_STATE
      android.permission.READ_EXTERNAL_STORAGE
      android.permission.READ_MEDIA_AUDIO
      android.permission.READ_MEDIA_VIDEO
      android.permission.ACCESS_MEDIA_LOCATION
      android.permission.FOREGROUND_SERVICE
      android.permission.USE_FULL_SCREEN_INTENT
      android.permission.RECORD_AUDIO
      com.android.providers.tv.permission.WRITE_EPG_DATA
      android.permission.POST_NOTIFICATIONS
      android.permission.ACCESS_WIFI_STATE
      android.permission.FOREGROUND_SERVICE_MEDIA_PLAYBACK
      android.permission.READ_PHONE_STATE
      android.permission.WRITE_SETTINGS
      android.permission.WRITE_EXTERNAL_STORAGE
      com.android.providers.tv.permission.READ_EPG_DATA
      me.aap.fermata.auto.dear.google.why.DYNAMIC_RECEIVER_NOT_EXPORTED_PERMISSION
      android.permission.MODIFY_AUDIO_SETTINGS
      android.permission.CAMERA
      android.permission.ACCESS_COARSE_LOCATION
      android.permission.ACCESS_FINE_LOCATION
    install permissions:
      android.permission.WRITE_SETTINGS: granted=false
      android.permission.MODIFY_AUDIO_SETTINGS: granted=true
      android.permission.MANAGE_EXTERNAL_STORAGE: granted=false
      android.permission.SYSTEM_ALERT_WINDOW: granted=false
      android.permission.FOREGROUND_SERVICE: granted=true
      android.permission.INTERNET: granted=true
      me.aap.fermata.auto.dear.google.why.DYNAMIC_RECEIVER_NOT_EXPORTED_PERMISSION: granted=true
      android.permission.USE_FULL_SCREEN_INTENT: granted=true
      android.permission.ACCESS_NETWORK_STATE: granted=true
      android.permission.DISABLE_KEYGUARD: granted=true
      android.permission.REQUEST_DELETE_PACKAGES: granted=true
      androidx.car.app.ACCESS_SURFACE: granted=true
      android.permission.ACCESS_WIFI_STATE: granted=true
      android.permission.REQUEST_INSTALL_PACKAGES: granted=false
      androidx.car.app.NAVIGATION_TEMPLATES: granted=true
      android.permission.WAKE_LOCK: granted=true
    User 0: ceDataInode=621419 installed=true hidden=false suspended=false distractionFlags=0 stopped=false notLaunched=false enabled=0 instant=false virtual=false
      overlay paths:
        /product/overlay/NavigationBarModeGestural/NavigationBarModeGesturalOverlay.apk
        /data/resource-cache/com.android.systemui-neutral-dKd9.frro
        /product/overlay/IconShapeRoundedRect/IconShapeRoundedRectOverlay.apk
        /data/resource-cache/com.android.systemui-accent-ouHF.frro
      legacy overlay paths:
        /product/overlay/NavigationBarModeGestural/NavigationBarModeGesturalOverlay.apk
        /product/overlay/IconShapeRoundedRect/IconShapeRoundedRectOverlay.apk
      gids=[3003]
      runtime permissions:
        android.permission.ACCESS_FINE_LOCATION: granted=false, flags=[ USER_SENSITIVE_WHEN_GRANTED|USER_SENSITIVE_WHEN_DENIED]
        android.permission.READ_EXTERNAL_STORAGE: granted=true, flags=[ USER_SET|USER_SENSITIVE_WHEN_GRANTED|USER_SENSITIVE_WHEN_DENIED|RESTRICTION_INSTALLER_EXEMPT]
        android.permission.ACCESS_COARSE_LOCATION: granted=false, flags=[ USER_SENSITIVE_WHEN_GRANTED|USER_SENSITIVE_WHEN_DENIED]
        android.permission.READ_PHONE_STATE: granted=false, flags=[ USER_SENSITIVE_WHEN_GRANTED|USER_SENSITIVE_WHEN_DENIED]
        android.permission.CAMERA: granted=false, flags=[ USER_SENSITIVE_WHEN_GRANTED|USER_SENSITIVE_WHEN_DENIED]
        android.permission.WRITE_EXTERNAL_STORAGE: granted=false, flags=[ USER_SENSITIVE_WHEN_GRANTED|USER_SENSITIVE_WHEN_DENIED|RESTRICTION_INSTALLER_EXEMPT]
        android.permission.RECORD_AUDIO: granted=false, flags=[ USER_SENSITIVE_WHEN_GRANTED|USER_SENSITIVE_WHEN_DENIED]
        android.permission.ACCESS_MEDIA_LOCATION: granted=true, flags=[ USER_SET|USER_SENSITIVE_WHEN_GRANTED|USER_SENSITIVE_WHEN_DENIED]
    User 11: ceDataInode=0 installed=false hidden=false suspended=false distractionFlags=0 stopped=true notLaunched=true enabled=0 instant=false virtual=false
      overlay paths:
        /data/resource-cache/com.android.systemui-neutral-dKd9.frro
        /product/overlay/IconShapeRoundedRect/IconShapeRoundedRectOverlay.apk
        /data/resource-cache/com.android.systemui-accent-ouHF.frro
      legacy overlay paths:
        /product/overlay/IconShapeRoundedRect/IconShapeRoundedRectOverlay.apk
      gids=[3003]
      runtime permissions:
        android.permission.ACCESS_FINE_LOCATION: granted=false
        android.permission.READ_EXTERNAL_STORAGE: granted=false, flags=[ APPLY_RESTRICTION]
        android.permission.ACCESS_COARSE_LOCATION: granted=false
        android.permission.READ_PHONE_STATE: granted=false
        android.permission.CAMERA: granted=false
        android.permission.WRITE_EXTERNAL_STORAGE: granted=false, flags=[ APPLY_RESTRICTION]
        android.permission.RECORD_AUDIO: granted=false
        android.permission.ACCESS_MEDIA_LOCATION: granted=false