search

gujjwal00 / avnc

VNC Client for Android
GNU General Public License v3.0
600 stars 57 forks source link

mouse click not working on window minimize & maximize & close on Android 14 #203

Closed SuJiKiNen closed 7 months ago

SuJiKiNen commented 7 months ago

click on some window's minimize, & maximize & close not working,and some other buttons. not sure if this problem is device specific,I have updated my Motorola x40 system to android 14 recently,I used termux proot I tested avnc and multi-vnc both not working，only got lucky on realvnc。 if anyone encounter this, we can discuss details further.

gujjwal00 commented 7 months ago

Please try enabling click-delay in AVNC:

  1. Long-press on a server, and tap 'Edit'
  2. Tap on 'Advanced' to open more options
  3. Enable 'Send delayed click events'
  4. Save
SuJiKiNen commented 7 months ago

I were tried this,no lucky yet

gujjwal00 commented 7 months ago

Which VNC server & operating system are you using? Did mouse click work before upgrading to Android 14?

SuJiKiNen commented 7 months ago

vnc: Xvnc TigerVNC 1.12.0 - built 2023-01-06 16:01

OS:
PRETTY_NAME="Kali GNU/Linux Rolling"
NAME="Kali GNU/Linux"
VERSION_ID="2023.4"
VERSION="2023.4"
VERSION_CODENAME=kali-rolling
ID=kali
ID_LIKE=debian
HOME_URL="https://www.kali.org/"
SUPPORT_URL="https://forums.kali.org/"
BUG_REPORT_URL="https://bugs.kali.org/"
ANSI_COLOR="1;31"

avnc:2.2.3

kail from https://www.kali.org/docs/nethunter/nethunter-rootless/

yes,it works before upgrade.

gujjwal00 commented 7 months ago

That's really weird. I don't have an Android 14 device, but when I get some time I will try to setup an emulator to see if the issue can be reproduced there. Meanwhile, please help me pinpoint the exact scenario where this bug happens.

  1. Please go to Settings => Tools => Touch test. Tap/double tap a few times in gray area. Copy and paste the logs here.
  2. Have you tried different "Gesture styles" in AVNC? You can change gesture style from toolbar.
  3. Do you have any other device running VNC server (e.g a laptop) you can test with?
  4. Can you also post a screen recording of the issue? (including where click works and where it doesn't)
SuJiKiNen commented 7 months ago

1.touch_test_log.txt 2.still same 3.no 4.I need figure out a way to show mouse click event.

I use mouse to operate,select text is fine,while left click not working,I turn offed the mouse pass through,the mouse click on those icons works,but still has some wired behaviors,when I right click on desktop,it seems couldn't distinguish left click and right click. I think it's some kind of events been false interpreted,I googled and found this.https://www.xda-developers.com/surface-pro-9-1000-deal/ it says some keycodes mapping changed.does this help?

after some tests,I presumed that click also recognized as touch gesture,since I change gesture setting double tap to left click,then right click twice on desktop same as left clicked.

gujjwal00 commented 7 months ago

1.touch_test_log.txt

These logs indicate that you are using your finger to click on touch screen when testing. If you really used the mouse for this test, than that might explain the issue you are facing. Otherwise please test with the same mouse you use for VNC session.

I use mouse to operate,select text is fine,while left click not working,I turn offed the mouse pass through,the mouse click on those icons works,but still has some wired behaviors,when I right click on desktop,it seems couldn't distinguish left click and right click. I think it's some kind of events been false interpreted

Yes, when you disable mouse passthrough, AVNC cannot differentiate between left and right mouse button. You need to long-press with either button to trigger right click on server.

after some tests,I presumed that click also recognized as touch gesture,since I change gesture setting double tap to left click,then right click twice on desktop same as left clicked.

Yes.

SuJiKiNen commented 7 months ago

thought you were asking testing with fingers,here is log that simulating till swipe,all using mouse left button. touchlog.txt

gujjwal00 commented 7 months ago

Thanks. Looking at these logs, mouse click should work correctly. Let me investigate a bit more.

SuJiKiNen commented 7 months ago

here is video,it's something wrong if mouse over the right half screen,maybe some coordinates or screen width miscalculated,I am using desktop mode,something similar to a virtual secondary screen, all actions using mouse left button.

https://github.com/gujjwal00/avnc/assets/6788956/5e8de119-e721-4a78-8e06-2619bc5a2ee9

gujjwal00 commented 7 months ago

Thanks for the video @SuJiKiNen, that's really helpful.

I am attaching a debug APK here: app-debug.zip It will log a lot of details about input processing in AVNC. Please run this, and do as follows:

  1. Connect to the server
  2. Open Terminal in Kali Linux
  3. Try to click on the 'Close' button on top-right corner
  4. Open 'About Qt' dialog
  5. Try to click on the 'Ok' button
  6. Try to close the dialog
  7. Drag the dialog towards left area (as you did in the video)
  8. Close the dialog
  9. Disconnect from servers

Now go to Settings => Tools => Logs. Copy the logs, and pate here. Tip: You don't need to create TXT files for logs. It is already formatted as a <details> element, so you can simply paste it here.

SuJiKiNen commented 7 months ago

thanks for your quick response,here your are. log.txt

gujjwal00 commented 7 months ago

I think you missed the file upload

SuJiKiNen commented 7 months ago

I think you missed the file upload

sorry,I re-uploaded.

gujjwal00 commented 7 months ago

Sorry, for some reason your device is filtering the logs (some device do this). Hence logs added by AVNC are not available in the output.

Please use this APK to run the test I mention above and re-upload the logs. It will keep the logs in app memory without going through Android: app-debug.zip

Full logs might be also be available through ADB, so let me know if you have a computer to run ADB.

SuJiKiNen commented 7 months ago

sorry for the late reply,I been busy on weekend,here is the log.hope this helps. debug-log.txt

gujjwal00 commented 7 months ago

sorry for the late reply,I been busy on weekend

No problem at all :)

Going through the logs, its looks like an issue with your Android version itself. Lets see two snippets of logs:

When clicking at point [3630, 97], not working:

23:13:54: com.gaurav.avnc.ui.vnc.TouchHandler.onTouchEvent(TouchHandler.kt:47): MotionEvent { action=ACTION_DOWN, actionButton=0, id[0]=0, x[0]=3630.0, y[0]=97.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142868126, downTime=142868126, deviceId=56, source=0x3002, displayId=15, eventId=884513322 }
23:13:54: com.gaurav.avnc.ui.vnc.TouchHandler.handleStylusEvent(TouchHandler.kt:127): MotionEvent { action=ACTION_DOWN, actionButton=0, id[0]=0, x[0]=3630.0, y[0]=97.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142868126, downTime=142868126, deviceId=56, source=0x3002, displayId=15, eventId=884513322 }
23:13:54: com.gaurav.avnc.ui.vnc.TouchHandler.handleMouseEvent(TouchHandler.kt:88): passthrough: true, e: MotionEvent { action=ACTION_DOWN, actionButton=0, id[0]=0, x[0]=3630.0, y[0]=97.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142868126, downTime=142868126, deviceId=56, source=0x3002, displayId=15, eventId=884513322 }
23:13:54: com.gaurav.avnc.ui.vnc.TouchHandler.onTouchEvent(TouchHandler.kt:47): MotionEvent { action=ACTION_UP, actionButton=0, id[0]=0, x[0]=3630.0, y[0]=97.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=0, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142868279, downTime=142868126, deviceId=56, source=0x3002, displayId=15, eventId=809558004 }
23:13:54: com.gaurav.avnc.ui.vnc.TouchHandler.handleStylusEvent(TouchHandler.kt:127): MotionEvent { action=ACTION_UP, actionButton=0, id[0]=0, x[0]=3630.0, y[0]=97.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=0, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142868279, downTime=142868126, deviceId=56, source=0x3002, displayId=15, eventId=809558004 }
23:13:54: com.gaurav.avnc.ui.vnc.TouchHandler.handleMouseEvent(TouchHandler.kt:88): passthrough: true, e: MotionEvent { action=ACTION_UP, actionButton=0, id[0]=0, x[0]=3630.0, y[0]=97.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=0, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142868279, downTime=142868126, deviceId=56, source=0x3002, displayId=15, eventId=809558004 }

When clicking at point [1611, 568], working as expected:

23:14:25: com.gaurav.avnc.ui.vnc.TouchHandler.onTouchEvent(TouchHandler.kt:47): MotionEvent { action=ACTION_DOWN, actionButton=0, id[0]=0, x[0]=1611.0, y[0]=568.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142898595, downTime=142898595, deviceId=56, source=0x3002, displayId=15, eventId=428762296 }
23:14:25: com.gaurav.avnc.ui.vnc.TouchHandler.handleStylusEvent(TouchHandler.kt:127): MotionEvent { action=ACTION_DOWN, actionButton=0, id[0]=0, x[0]=1611.0, y[0]=568.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142898595, downTime=142898595, deviceId=56, source=0x3002, displayId=15, eventId=428762296 }
23:14:25: com.gaurav.avnc.ui.vnc.TouchHandler.handleMouseEvent(TouchHandler.kt:88): passthrough: true, e: MotionEvent { action=ACTION_DOWN, actionButton=0, id[0]=0, x[0]=1611.0, y[0]=568.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142898595, downTime=142898595, deviceId=56, source=0x3002, displayId=15, eventId=428762296 }
23:14:25: com.gaurav.avnc.ui.vnc.TouchHandler.onGenericMotionEvent(TouchHandler.kt:54): MotionEvent { action=ACTION_BUTTON_PRESS, actionButton=BUTTON_PRIMARY, id[0]=0, x[0]=1611.0, y[0]=568.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142898595, downTime=142898595, deviceId=56, source=0x2002, displayId=15, eventId=486625487 }
23:14:25: com.gaurav.avnc.ui.vnc.TouchHandler.onHoverEvent(TouchHandler.kt:59): MotionEvent { action=ACTION_BUTTON_PRESS, actionButton=BUTTON_PRIMARY, id[0]=0, x[0]=1611.0, y[0]=568.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142898595, downTime=142898595, deviceId=56, source=0x2002, displayId=15, eventId=486625487 }
23:14:25: com.gaurav.avnc.ui.vnc.TouchHandler.handleStylusEvent(TouchHandler.kt:127): MotionEvent { action=ACTION_BUTTON_PRESS, actionButton=BUTTON_PRIMARY, id[0]=0, x[0]=1611.0, y[0]=568.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142898595, downTime=142898595, deviceId=56, source=0x2002, displayId=15, eventId=486625487 }
23:14:25: com.gaurav.avnc.ui.vnc.TouchHandler.handleMouseEvent(TouchHandler.kt:88): passthrough: true, e: MotionEvent { action=ACTION_BUTTON_PRESS, actionButton=BUTTON_PRIMARY, id[0]=0, x[0]=1611.0, y[0]=568.0, toolType[0]=TOOL_TYPE_MOUSE, buttonState=BUTTON_PRIMARY, classification=NONE, metaState=0, flags=0x0, edgeFlags=0x0, pointerCount=1, historySize=0, eventTime=142898595, downTime=142898595, deviceId=56, source=0x2002, displayId=15, eventId=486625487 }
23:14:25: com.gaurav.avnc.ui.vnc.Dispatcher$AbstractMode.doButtonDown(Dispatcher.kt:193): button: Left, p:PointF(1611.0, 568.0)
23:14:25: com.gaurav.avnc.vnc.Messenger.sendPointerButtonDown(Messenger.kt:60): button: Left, p:PointF(788.3334, 315.55557), pointerButtonMask:0
23:14:25: com.gaurav.avnc.vnc.VncClient.sendPointerEvent(VncClient.kt:171): Sending pointer event: 1 at [788 , 315]

Android reports full-fledged mouse events via onGenericMotionEvent() call. These are the lines with action=ACTION_BUTTON_PRESS. You will notice that in first case such lines are missing, i.e. Android is not generating them for some reason. So no events is sent by AVNC to the server. But when you click in left half of the screen, events are generated as expected, and click works.

Because MultiVNC is also affected, and this only started happening after Android update, I think AVNC might not be the issue here.

To confirm this theory, we need to test other apps. If you have any other app which uses mouse right click differently (e.g. right click opens a menu), you can test to see if it has the same issue in different halves of the screen. One such app might be the web browser.

SuJiKiNen commented 7 months ago

I tested some office document apps,works fine.

after some debug testing,click on right half screen, AVNC only receives gesture events.

gujjwal00 commented 7 months ago

Well, that's a bummer 😕. Only thing left now is to see if I can reproduce the bug on my side. Please let me know some details about you setup:

I will let you know if/when I find something. In the meantime, you can disable mouse pass through.

SuJiKiNen commented 7 months ago

  1. Android flavour/skin (e.g. MIUI, OneUI etc.) MYUI6.0

  2. Device details (screen size, model) 2400X1800,XT2301-5

  3. Any special software you are using to enable desktop mode or taskbar Ready For Assistant on PC side through WiFi.

  4. Screen size on server this match client PC screen,1920X1080

  5. Anything else worth noting mouse click start failed if AVNC size is bigger than half screen width,

    ----------half--------|--fail---|
half = scrren width / 2
0 < fail width < screen width /2

I have full dev setup now,but don't know where to start

gujjwal00 commented 7 months ago

2. Device details (screen size, model) 2400X1800,XT2301-5

That's odd. In my https://github.com/gujjwal00/avnc/issues/203#issuecomment-1914727186 above, you can see that the first click happens at coordinate [x= 3630, y= 97], but this screen size is much smaller than that.

I have full dev setup now,but don't know where to start

That great! How comfortable are you with Android Studio, or IDEs in general?

SuJiKiNen commented 7 months ago

that's another pc .

I used Android Studio twice before,a lot of things need to learn. source code is well documented,but I lack professional knowledge. could you give me some hints how to debug if motionEvent is consumed by some activities or not delivered as intend by android itself.

gujjwal00 commented 7 months ago

that's another pc .

I have a feeling this is the root cause, but lets wait and see.

I used Android Studio twice before,a lot of things need to learn. source code is well documented,but I lack professional knowledge. could you give me some hints how to debug if motionEvent is consumed by some activities or not delivered as intend by android itself.

Primary point of interest in AVNC is the TouchHandler class. This is where all MotionEvents are received by three functions: onTouchEvent(), onGenericMotionEvent(). onHoverEvent(). onGenericMotionEvent() is the one which handles mouse clicks.

But first, lets start with something simple. We can test MotionEvent delivery with a brand new project in Android Studio. This will be easier to start for you, and it will rule out any bug in AVNC.

I am using Android Studio HedgeHog 2023.1.1. Follow these steps:

  1. Create new project
  2. In New Project dialog, select Empty View Activity, and click Next, and then Finish
  3. Let it sync/build, and then run it on you device to make sure everything woks as expected.
  4. In activity_main.xml, add android:id="@+id/root" to the ConstraintLayout. Like this: 
    <androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity"
android:id="@+id/root">

  5. In MainActivity.kt, add touch event listeners to root view, like this:

    class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)

    /*Add event listeners*/
    val root = findViewById<View>(R.id.root)
    root.setOnTouchListener { _, event ->
        Log.d("EventTest", "TouchEvent: $event")
        true
    }

    root.setOnGenericMotionListener { _, event ->
        Log.d("EventTest", "GenericEvent: $event")
        true
    }

    /* Hover generates a lot of events. Uncomment this to log all mouse movements
    root.setOnHoverListener { _, event ->
        Log.d("EventTest", "HoverEvent: $event")
        true
    }*/
}
}
  6. Now run the app, and click on different parts of the app.

You can open the Logcat window to see logged events, or you can set breakpoints on listeners and debug the app. Our primary interest is the GenericEvent callback.

SuJiKiNen commented 7 months ago

Thank you very very very much. now I found the bug may come from the desktop mode itself. here is log that click left and right button on left half screen,and then the opposite. eventTest.log I guess other apps works,because they test mouse click by [event.getButtonState()](https://developer.android.com/reference/android/view/MotionEvent#getButtonState())

Mouse and stylus buttons can be retrieved using [getButtonState()](https://developer.android.com/reference/android/view/MotionEvent#getButtonState()). It is a good idea to check the button state while handling ACTION_DOWN as part of a touch event. The application may choose to perform some different action if the touch event starts due to a secondary button click, such as presenting a context menu.

how do you think?

SuJiKiNen commented 7 months ago

I found this is not able to tell which button been released.

gujjwal00 commented 7 months ago

I found this is not able to tell which button been released.

That's the main reason. But if you look through the issues in this repo, you will find many issues involving touch input. Many devive manufacturers mess with stock Android behaviour, so current implementation in AVNC has seen many revisions, and handles a lot of wierd devices.

Best choice here is to report this issue to Motorola and get the root cause fixed by them, instead of finding a workaround which might break with the next upgrade.

SuJiKiNen commented 7 months ago

yes,that's best what we can do.IMO,android itself also a bit messy.

gujjwal00 commented 7 months ago

IMO,android itself also a bit messy.

Yeah, in some areas, developing for Android is just too messy. What's more frustrating is that I, as a developer, have no way to reproduce such bugs. Every manufacturer mess with stock behavior, and these ROM's only run on specific devices.

Anyway, I am closing this bug. Hopefully Motorola listens to your feedback.