ESP32 BT-HID Mouse

[diegoasth]

I am trying to control my android tablet with an ESP32 via BLE. I have an app for musical chords and sheet music and I would like to turn pages on the app just pressing a button on the ground with my feet, while playing guitar. For certain features of this app i have to use mouse scrolls (features controlled by arrows down/up are ok thanks to this library)

I tried to reproduce this video https://www.youtube.com/watch?v=zgW9Hc1bgiY by @chegewara and studied this code https://github.com/asterics/esp32_mouse_keyboard/blob/newBLE/components/nkolban_BLE/HID_kbdmousejoystick.cpp and came to my code attached.

Well, it is not working.. any ideas? Thank you

HID_mouse.ino.txt

[chegewara]

This is good material to study: https://eleccelerator.com/tutorial-about-usb-hid-report-descriptors/

[diegoasth]

It seems to work! thanks for the guidance @chegewara . Here is my code. BLE_Mouse.ino.txt

[senthilkumarh]

It seems to work! thanks for the guidance @chegewara . Here is my code. BLE_Mouse.ino.txt

HI, I used the sketch you attached BLE_Mouse.ino and it is working fine with android phone but its not working when connected to windows10. Please can you help on this

[ShubhamJain7]

@chegewara @senthilkumarh I am using the following report map to implement a BLE mouse. `

 USAGE_PAGE(1),       0x01,
  USAGE(1),         0x02,
   COLLECTION(1),     0x01,
   REPORT_ID(1),      0x01,
   USAGE(1),          0x01,
   COLLECTION(1),     0x00,
   USAGE_PAGE(1),     0x09,
   USAGE_MINIMUM(1),    0x1,
   USAGE_MAXIMUM(1),    0x3,
   LOGICAL_MINIMUM(1),  0x0,
   LOGICAL_MAXIMUM(1),  0x1,
   REPORT_COUNT(1),     0x3,
   REPORT_SIZE(1),      0x1,
   0x80|0x01,           0x2,    // (Data, Variable, Absolute), ;3 button bits
   REPORT_COUNT(1),     0x1,
   REPORT_SIZE(1),      0x5,
   0x80|0x01,           0x1,    //(Constant), ;5 bit padding
   USAGE_PAGE(1),       0x1,    //(Generic Desktop),
   USAGE(1),            0x30,
   USAGE(1),            0x31,
   LOGICAL_MINIMUM(1),  0x81,
   LOGICAL_MAXIMUM(1),  0x7f,
   REPORT_SIZE(1),    0x8,
   REPORT_COUNT(1),   0x2,
   0x80|0x01,        0x6,    //(Data, Variable, Relative), ;2 position bytes (X & Y)
   END_COLLECTION(0),
  END_COLLECTION(0)`

How do I modify the code: uint8_t msg[] = {0x00, 0x00, 0x00, 0x00}; to move the cursor to the coordinates (x,y) on the screen? (x and y are of variables of type int16_t)

[chegewara]

Hi, you need to modify report map a bit. Try to play with those values:

USAGE(1),            0x30,
   USAGE(1),            0x31,
   LOGICAL_MINIMUM(1),  0x81, // <-- 0x00
   LOGICAL_MAXIMUM(1),  0x7f, // <-- 0xff
   REPORT_SIZE(1),    0x8,
   REPORT_COUNT(1),   0x2,
   0x80|0x01,        0x6,    //(Data, Variable, Relative), ;2 position bytes (X & Y) <--- change to absolute instead of relative

This is starting point.

[ShubhamJain7]

Hi, you need to modify report map a bit. Try to play with those values:

USAGE(1),            0x30,
   USAGE(1),            0x31,
   LOGICAL_MINIMUM(1),  0x81, // <-- 0x00
   LOGICAL_MAXIMUM(1),  0x7f, // <-- 0xff
   REPORT_SIZE(1),    0x8,
   REPORT_COUNT(1),   0x2,
   0x80|0x01,        0x6,    //(Data, Variable, Relative), ;2 position bytes (X & Y) <--- change to absolute instead of relative

This is starting point.

Thanks for the reply @chegewara ! I tried experimenting with the values you suggested but couldn't achieve what I wanted and I don't know how to change it to absolute in place of relative.

I'm trying to implement the Mouse.h library(https://github.com/arduino-libraries/Mouse/blob/master/src/Mouse.cpp) for ESP32 and have functions for moving, clicking and releasing clicks. But the report map in the link is different from yours and unfortunately, I don't understand how report maps work :(

[chegewara]

https://eleccelerator.com/usbdescreqparser/

0x80|0x01, 0x2, // <-- absolute value

[ShubhamJain7]

@chegewara, I changed the report map as you said but it had no effect. If I send constant values, say uint8_t msg[] = {0x00, 0x0A, 0x0A, 0x00}; the mouse pointer moves 10 pixels to the right and down in each iteration of the loop. I what I want is for it to stay at the location (10,10).

Sorry if my doubts and issues are silly, I'm just a beginner.

[chegewara]

Hi, i know it works because it was used and tested in this code by friend of mine @markingle: https://github.com/markingle/Assist_IoT_ESP32_LipSync/blob/master/Lip_Sync_v2.ino#L156

The only think is that you can use 0xA to point exactly pixel with coordinate 10, you have to use map function to do it.

[xcarcelle]

@ShubhamJain7 : have you been able to have absolute coordinates to work on your tests ? Which host devices have you tested ? Willing to do tests on android/ios if needed.

[xcarcelle]

@chegewara : I have tested some modifications on @diegoasth code to try absolute mouse moveto. Simple tests results :

• relative mouse code = android and ios14 : mouse detected and moving relatively
• abs mouse code = android : mouse detected and moving relatively, ios14 : mouse detected but not moving I am missing a part on "map function" to have the abs mouse (x, y) work on this simple code. Thanks for your time and help on this review !

es32_ble_mouse_simple_abs.ino.txt es32_ble_mouse_simple_rel.ino.txt

[chegewara]

Ok, i dont know if its helpful, but today i tested some code and appears that android, at least android 10 on my samsung S9+, is not working good with absolute mouse cursor position. The same code with relative works perfectly. In other hand i tested the same code with absolute position on linux and it works exactly what it should, so its not esp32 problem.

[xcarcelle]

@chegewara thanks for your reply. When you say "it is not working good" with Android 10, do you have at least the cursor of the mouse or no detection at all of the absolute mouse as a peripheral device ? There is something when absolute mouse is activated in the HID Descriptor that seems not to be handled correctly by Anrdoid/iOS maybe on the way the report is sent (at initial handshake). Quite difficult to debug if it the case w/o HID logs on the host side. However some usb touchscreen seem to be handled correctly (I don't have a product name to quote) with specific HID Descriptor and report sending at handshake which could help to debug this features in absolute mode. No issue on the esp32 side. Thanks again for your help and test !

[chegewara]

Thats the case, android is recognizing as hid device, because is auto-reconnecting to it, but no cursor on screen. I can see cursor with the same code when i change absolute X/Y to relative value.

[xcarcelle]

yes same for me relative works fine but once absolute is triggered no mouse. Maybe a rooted android device with HCI debugging tool could help. Also a working usb touchscreen could be great. Also this implementation seems interesting (not tested yet) : https://gist.github.com/mbt28/406bdf15a248029c774085832c7c0c0c

[chegewara]

Yes, that code is really interesting. I never been trying to do touch screen, but worth to save for future.

[chegewara]

I know @Nesh108 solved the problem by using digitizer, at least on android. Maybe he can share report map with us.

[Nesh108]

This is the report map working on Android and absolute 16 bit coordinates:

0x05, 0x0D,        // Usage Page (Digitizer)
0x09, 0x02,        // Usage (Pen)
0xA1, 0x01,        // Collection (Application)
0x85, 0x01,        //   Report ID (1)
0x05, 0x0D,        //   Usage Page (Digitizer)
0x09, 0x01,        //   Usage (Digitizer)
0xA1, 0x00,        //   Collection (Physical)
0x09, 0x42,        //     Usage (Tip Switch)
0x09, 0x32,        //     Usage (In Range)
0x15, 0x00,        //     Logical Minimum (0)
0x25, 0x01,        //     Logical Maximum (1)
0x75, 0x01,        //     Report Size (1)
0x95, 0x03,        //     Report Count (3)
0x81, 0x02,        //     Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)
0x75, 0x05,        //     Report Size (5)
0x95, 0x01,        //     Report Count (1)
0x81, 0x01,        //     Input (Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position)
0x05, 0x01,        //     Usage Page (Generic Desktop Ctrls)
0x09, 0x30,        //     Usage (X)
0x16, 0x00, 0x00,  //     Logical Minimum (0)
0x36, 0x00, 0x00,  //     Physical Minimum (0)
0x26, 0x80, 0x07,  //     Logical Maximum (1920)
0x46, 0x80, 0x07,  //     Physical Maximum (1920)
0x09, 0x31,        //     Usage (Y)
0x16, 0x00, 0x00,  //     Logical Minimum (0)
0x36, 0x00, 0x00,  //     Physical Minimum (0)
0x26, 0x38, 0x04,  //     Logical Maximum (1080)
0x46, 0x38, 0x04,  //     Physical Maximum (1080)
0x75, 0x10,        //     Report Size (16)
0x95, 0x02,        //     Report Count (2)
0x81, 0x22,        //     Input (Data,Var,Abs,No Wrap,Linear,No Preferred State,No Null Position)
0xC0,              //   End Collection
0xC0,              // End Collection

// 72 bytes

This is how to use it:

          uint8_t x1 = 0x50;
          uint8_t x2 = 0x03;
          uint8_t y1 = 0x05;
          uint8_t y2 = 0x05;

          unsigned char buffer[] = {0x01, 0x00, x1 & 0x00FF, (x2 & 0xFF00) >> 8, y1 & 0x00FF, (y2 & 0xFF00) >> 8};

          inputMouse->setValue(buffer,sizeof(buffer));
          inputMouse->notify();

Now trying to get it to work on iOS, this solution doesn't work there at the moment.

[TheBricktop]

for some reason none of the sketch here work for me either on android (s8 android 9.0), or windows10 ESP32 Wroom, didnt do much but set up the latest esp32 lib for arduino and pasted some of the sketches of @xcarcelle and @chegewara, both the abs and rel versions of mouse but im not seeing any movement of cursor on the host machines. No errors in the compilation serial reads the messages from esp so the program runs ok but board reports is not processed or registered as a HID device.