Feature Request - absolute mouse

jrsmile commented 4 years ago

Would it be possible to adapt the mouse hid class to send absolute coordinates like a touchscreen?

benjaminaigner commented 4 years ago

Dear @jrsmile , should be possible. I cannot estimate if and when we will include this, joystick support and stable connections have a higher priority. If you want to try it on your own: This is the HID report map: https://github.com/asterics/esp32_mouse_keyboard/blob/f7ab88ae7a9ba4b450296a1b7970a30fa985b622/main/ble_hid/hid_device_le_prf.c#L44 There we need to replace the relative mouse by an absolute positioned. Here is an example for this report map: https://forums.obdev.at/viewtopic.php?t=2559

jrsmile commented 4 years ago

i have seen a map here: HERE unfortunately i know to little to modify the code the correct way. would the link help with the implementation?

ChrisVeigl commented 4 years ago

hi, we added absolute mouse position and joystick reports to our HID-actuator several years ago - it worked. mybe this is helpful:

https://github.com/asterics/AsTeRICS/blob/master/CIMs/HID_actuator/Mouse_Keyboard_Joystick/Descriptors.c https://github.com/asterics/AsTeRICS/blob/master/CIMs/HID_actuator/Mouse_Keyboard_Joystick/Descriptors.h

benjaminaigner commented 4 years ago

@ChrisVeigl , @jrsmile THX for the suggestions, based on this I will implement an additional report with an absolute mouse.

jrsmile commented 4 years ago

thank you very much, looking forward to it. i will use it to build a firmware test tool. best possible outcome would be a keyboard,mouse,touchscreen device and a ble serial device. being able to test with a mobile phone as gui via ble a machine that has a bluetooth dongle attached is my target :-) i wonder if pairing with two devices is possible. if not usb serial will be suficient too.

benjaminaigner commented 4 years ago

Ah, sounds nice! I've started a new branch with the absolute mouse included. If there is time this week, I will test it.

Regarding the serial device: This is a feature we won't implement by ourselves, but we could need that :-)

Pairing with multiple devices: Pairing is possible, connecting should be too. I think it is necessary to store different HID connection IDs for multiple devices, but this is also something thats not on our agenda.

benjaminaigner commented 4 years ago

@jrsmile

In the branch "absoluteMouse", there is a basic implementation of an absolute mouse report. I've some input on Android, but I think it is not completely correct.

Could you please test it and report back?

zuozhehao commented 4 years ago

Is there any progress change.

benjaminaigner commented 4 years ago

@zuozhehao

I have a branch with the absolute mouse descriptor for testing. Please try it out and report back. If everything is working, I'll merge the changes and make it available via config files.

xcarcelle commented 4 years ago

@benjaminaigner @zuozhehao thanks for the absolute mouse branch/progress and any help needed on test or code i will try to support this issue.

I have tested the last branch commit on an Android tablet running 9.0 and it acts weird (like keys from a keyboard pushed very fast) when trying the absolute command from the uart in esp-idf (running make monitor flash)
The mouse click also does not seem to work Do you have some unit tests (devices, os, test) you want to run to validate this current state ?

benjaminaigner commented 4 years ago

Dear all, I'm sorry this feature takes so long. Absolute mouse is not our top priority & due to COVID my workload regarding lectures is rising dramatically.

Still, I try my best to support you and see what we can do about the absolute mouse.

Current status: @xcarcelle I've seen the same behaviour on my Android device.

I wanted to try my "real" absolute mouse (touchscreen) on iOS for @zuozhehao , but iPads do not accept USB accessory with 100mA current. Unfortunately I have no USB-C OTG cable available to test it on Android.

I will keep you updated, today I can spare some time for further testing.

benjaminaigner commented 4 years ago

OK, I tested following absolute HID descriptors:

https://www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with

Variant 0: works on Debian, but not on iOS / Android Variant 1: works on Android, but not on Debian / iOS Variant 2: didn't get it to work on all platforms

https://www.schoeldgen.de/avr/

(USB Interface for Tektronix 4957 tablet using V-USB) Works on Debian, not on iOS/Android (most promising for iOS, because the Assistive Touch indicator went from grey to black)

Next try: remove all other reports except the absolute mouse, maybe this helps for iOS. Does anybody has an alternative absolute mouse descriptor to test?

xcarcelle commented 4 years ago

@benjaminaigner thanks for the update, i will test android and ios on multiple devices tomorrow. How do you test variant0/1/2 from https://www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with with your absolute branch ? Can you confirm the place to replace HID Descriptors. For Tektronix 4957, can I test something at this point with an iOS device ? Keeping posted to further instructions.

xcarcelle commented 4 years ago

@benjaminaigner I have tried replacing the "touch_device=1" HID Descriptor from codeproject into this block : HID Descriptor Mouse Absolute but it did not work for abs move on Android (8.0 and 9.0) tablets and iOS. Could you tell me how did you replace the HID Descriptor in your code so I can follow-up tests ? Regards.

benjaminaigner commented 4 years ago

Dear @xcarcelle ,

for testing, I replaced following parts accordingly (in master branch): https://github.com/asterics/esp32_mouse_keyboard/blob/ffc6d08332428d8c128278c1fc48e61d6619c04b/main/esp_hidd_prf_api.c#L135 change the bytestuffing for the mouse report here (some HID descriptors require more than 5 Bytes; change HID_MOUSE_IN_RPT_LEN accordingly)

Place the HID report map here: https://github.com/asterics/esp32_mouse_keyboard/blob/ffc6d08332428d8c128278c1fc48e61d6619c04b/main/hid_device_le_prf.c#L145 Please be careful to place the report ID in any report map to be tested: 0x85, 0x03, // Report Id (3)

For testing, I placed this source into the main file:

//at the beginning of uart_console_task:
static uint16_t absX = 0;
static uint16_t absY = 0;

//in the switch(character) block
case '0':
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: l");
    esp_hidd_send_mouse_value(hid_conn_id,(1<<0),absX,absY,0);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '1':
    absX = 0;
    absY = 0x2710;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '2':
    absX = 0x2710/2;
    absY = 0x2710;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '3':
    absX = absY = 0x2710;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '4':
    absX = 0;
    absY = 0x2710/2;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '5':
    absX = absY = 0x2710 / 2;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '6':
    absX = 0x2710;
    absY = 0x2710/2;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '7':
    absX = absY = 0;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '8':
    absX = 0x2710/2;
    absY = 0;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;
case '9':
    absX = 0x2710;
    absY = 0;
    ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
    esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
    break;

Yesterday night, I tried my 13" touchscreen with Android, connected via USB. It works flawlessly for touch input, but there is no cursor available. Nevertheless, I cannot try this device with iOS, because the iPad refuses to use this USB device ("accessory requires too much power"). It would be very interesting to see if this device works with iOS (would be a good start for our own device).

xcarcelle commented 4 years ago

Dear @benjaminaigner , thks a lot for your update, first the step i used to test your proposal :

using the master branch
replacing the HID Descriptor with the one from codeproject (TOUCHDEVICE=1) as is
not changing the HID_MOUSE_IN_RPT_LEN at this point
adding the console command for abs move Then the test using (esp32 board + Huawey Mediapad T5 Android 8.0) is that the mouse moves but only in a small up-left portion of the tablet screen, the move(0, 0) seems to work but the other values does not move on the way to half/full-screen position. I did sent you a video of the current test. Will try with iOS tablet over BLE. Do I miss something for the half/full screen moveto values in the code to enable the full screen ? Regards.

xcarcelle commented 4 years ago

@benjaminaigner : I was wondering if you had more tests with master or absolute branch to test the different HID Descriptors. I want to test different descriptors with different devices but I want to be sure to replicate your methodology to push significant results. Cheers.

benjaminaigner commented 4 years ago

Dear @xcarcelle I'm sorry that my reply time is so high, currently we have a high workload of preparing online classes for Corona-safe lectures :-).

Regarding your question with your test:

absY = 0x2710/2; should be half the screen size, BUT: this is only valid, if you use 0x2710 as value in the HID descriptor (Logical Maximum (10000))
In absolute mouse settings, the host device maps the logical max value to the current screen size. So if you send the logical maximum value as coordinate, the mouse should be in the other corner (X/Y: max-> bot /right corner; X max, Y 0 -> top/right corner; and so on).

I had still no time until now to extract the HID descriptor of my USB-Touchscreen, which works on Android & Linux (can't be tested on iOS because the Lightning to USB adapter does not provide sufficient current).

xcarcelle commented 4 years ago

@zuozhehao : have you been able to do some tests on your side with android/ios ?

xcarcelle commented 4 years ago

Dear @benjaminaigner I guess the workload is crazy w/ the current Covid situation and remote session of classes, good luck ! I will continue more tests tomorrow and I have a set-up with ios and Lightning-to-USB_and_power cable. Which model of USB-Touchscreen are you trying ? I wish I could test on Android first, if you have some ready-to-test code I am willing to build/test it. Cheers and good luck !

benjaminaigner commented 4 years ago

Dear @xcarcelle , I finally got time to dump the HID descriptor from the touchscreen. It is a Iltek ICI2511 controller, USB: 222a:0001 I was quite surprised, this HID map is really big. touchscreen_HIDMap.txt

I've uploaded the raw dump from USBlyzer, which needs re-formating into a C-array. If i can spend time, I will put this report into our project (please be patient, could take a few days).

benjaminaigner commented 4 years ago

At least I found the time to put into a C-array... If anybody can test it, I would be really thankful. But be careful, this report descriptor needs heavy reformating to put the data in the necessary format (report IDs, mouse clicks & X/Y data). reportMapDigitizer.txt

benjaminaigner commented 4 years ago

This is the corresponding dump of HID data:

Press 1 finger
Release 1 finger
Press 3 fingers
Release 3 fingers

As this seems to be a multitouch display (1B for button + 4B for X/Y repeat 10x times), the HID report map is a little bit messed up. USBlyzer cannot assign the single touch values correspondingly (everything is mapped many times to the first touch input). Contact count is used to set the number of input points. I don't know what the scan time means, but it seems to be an incrementing counter...

Sry for the format of this file, quick & dirty copy'n'paste...

touchscreen_dumps.txt

benjaminaigner commented 4 years ago

Following problems with this HID report map: BLE MTU limits the packet sizes for HID payload to 20B. The original frame size was 64B.

I've tried to strip down this report map to following result, but currently not functioning on Android (if anybody can spot the error, I would be really happy):

0x05, 0x0D, // Usage Page (Digitizer)
0x09, 0x04, // Usage (Touch Screen)
0xA1, 0x01, // Collection (Application)
    0x85, 0x03, // Report ID (3)
    0x09, 0x22, // Usage (Finger)
    0xA1, 0x02, // Collection (Logical)
        0x05, 0x0D, // Usage Page (Digitizer)
        0x95, 0x01, // Report Count (1)
        0x75, 0x06, // Report Size (6)
        0x09, 0x51, // Usage (Contact Identifier)
        0x15, 0x00, // Logical Minimum (0)
        0x25, 0x3F, // Logical Maximum (63)
        0x81, 0x02, // Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
        0x09, 0x42, // Usage (Tip Switch)
        0x25, 0x01, // Logical Maximum (1)
        0x75, 0x01, // Report Size (1)
        0x95, 0x01, // Report Count (1)
        0x81, 0x02, // Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
        0x75, 0x01, // Report Size (1)
        0x95, 0x01, // Report Count (1)
        0x81, 0x03, // Input (Cnst,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
        0x05, 0x01, // Usage Page (Generic Desktop)
        0x75, 0x10, // Report Size (16)
        0x55, 0x0E, // Unit Exponent (-2)
        0x65, 0x11, // Unit (SI Lin: Length (cm))
        0x09, 0x30, // Usage (X)
        0x26, 0x00, 0x40, // Logical Maximum (16384)
        0x35, 0x00, // Physical Minimum (0)
        0x46, 0x15, 0x0C, // Physical Maximum (3093)
        0x81, 0x42, // Input (Data,Var,Abs,NWrp,Lin,Pref,Null,Bit)
        0x09, 0x31, // Usage (Y)
        0x26, 0x00, 0x40, // Logical Maximum (16384)
        0x46, 0xCB, 0x06, // Physical Maximum (1739)
        0x81, 0x42, // Input (Data,Var,Abs,NWrp,Lin,Pref,Null,Bit)
    0xC0,  // End Collection
    0x05, 0x0D, // Usage Page (Digitizer)
    0x09, 0x56, // Usage (Scan Time)
    0x55, 0x00, // Unit Exponent (0)
    0x65, 0x00, // Unit (None)
    0x27, 0xFF, 0xFF, 0xFF, 0x7F, // Logical Maximum (2147483647)
    0x95, 0x01, // Report Count (1)
    0x75, 0x20, // Report Size (32)
    0x81, 0x02, // Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
    0x09, 0x54, // Usage (Contact Count)
    0x25, 0x7F, // Logical Maximum (127)
    0x95, 0x01, // Report Count (1)
    0x75, 0x08, // Report Size (8)
    0x81, 0x02, // Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
0xC0,  // End Collection

I've used this output function, now HID_MOUSE_IN_RPT_LEN is 10B.

        static uint32_t scantime = 0;
        uint8_t buffer[HID_MOUSE_IN_RPT_LEN];

        buffer[0] = 0x40 | (mouse_button-1);   // Buttons
        buffer[1] = mickeys_x & 0xFF;           // X low byte
        buffer[2] = (mickeys_x>>8) & 0xFF;       // X high byte
        buffer[3] = mickeys_y & 0xFF;           // Y low byte
        buffer[4] = (mickeys_y>>8) & 0xFF;       // Y high byte*/

        //scantime
        buffer[5] = scantime & 0xFF;
        buffer[6] = (scantime>>8) & 0xFF;
        buffer[7] = (scantime>>16) & 0xFF;
        buffer[8] = (scantime>>24) & 0xFF;

        buffer[9] = 0x01; //contact count, increase for multiple touch input

        scantime++;

xcarcelle commented 4 years ago

@benjaminaigner thanks a lot for all these informations ! I will give a try on iOS later today and feedback you on this HID Descriptor if I can implement this Descriptor/Report.

asterics / esp32_mouse_keyboard

Feature Request - absolute mouse #30