feature request: Absolute mode

street-grease-coder commented 3 years ago

Is there any possibility that there will be an absolute mode? That would be more useful and intuitive since otherwise I don't see a way around resetting the mouse position if it has accidentally 'run against a border', if that makes sense!

Also, sorry, I'm new to this field

T-vK commented 3 years ago

Maybe, but for this to work, you manually have to tell your ESP the screen resolution, screen orientation, cursor speed and cursor acceleration of the device you want to connect to and it also needs to know the current cursor position which can always change if you have a physical mouse connected as well or alternatively you can reset the mouse position (for example by moving it to the top left corner) before every move with expected side-effects and performance loss.

Something else that is being worked on is emulating touchscreen input: https://github.com/T-vK/ESP32-BLE-Mouse/issues/5

street-grease-coder commented 3 years ago

Ok, so from this I take you assume there is no feasible way to get parameters like screen resolution from any accessible HID library, I was kind of hoping that would be the case, since somehow logitech products seem to be able to do this (air mice). I guess I was hoping to not need to calculate XY (using accel settings etc) but to get it sent back over the BLE interface from the computer. Sad but the reset of the position will work as a workaround :) thanks

T-vK commented 3 years ago

I highly doubt there is a way to do this without additional software running on the target device. I could be wrong, but Logitech most likely uses a custom driver or an additional piece of software that you have to install on the target device in order for this to work. This software could be communicating with the mouse using a rfcomm serial connection. I think there has been a proof of concept on having a Bluetooth serial connection and a BLE server running at the same time: https://github.com/beegee-tokyo/ESP32-Weatherstation/blob/93079caa319eaeb1c237782bf348cafd96e6e3c1/docs/bt_serial.cpp

xcarcelle commented 3 years ago

@street-grease-coder : thanks for your interest on this issue and we did several tests enabling the absolute mode in HID Descriptor but neither ios14+ ("ble touchsreen" HID seems to be disabled since ios13.4 https://discussions.apple.com/thread/251720560) nor android8+ devices. It seems that using absolute HID Descriptor + uint16_t coordinates datatypes from working usb HID Descriptor is not enough and they might be an issue with the ble packets sent to the HID driver of the host ? Some product (like http://www.acrosscenter.com/) seems to have remote mouse control over (ble + app/driver). What is your target devices/os ?

@T-vK : thanks for your reply and surely having (spp + ble) at the same time could help sending from the host some calibration information such as the resolution and orientation. Could you explain where to indicate the screen orientation and resolution in the sp32 code ? I could try to modify these part to adjust some settings on the ble mouse.

T-vK commented 3 years ago

Could you explain where to indicate the screen orientation and resolution in the sp32 code ? I could try to modify these part to adjust some settings on the ble mouse.

No, the code has not been written yet. ... Thinking about how this would be implemented, I think it would only be necessary to know the cursor speed and acceleration.

But I'd imagine it to look somewhat like this:

#include <math.h> // required because of fmod

void moveTo(float newX, float newY, float speed = 1, float acceleration = 0, bool isLandscape = true, float initialX = 0, float initialY = 0) {
    if (!isLandscape) { // Swap x and y in portrait mode
        float tmp;
        tmp = newX;
        newX = newY;
        newY = tmp;
        tmp = initialX;
        initialX = initialY;
        initialY = tmp;
    }

    float currentX = initialX;
    float currentY = initialY;

    float maxStepSizeX = (initialX < newX) ? 127 : -128;
    float maxStepSizeY = (initialY < newY) ? 127 : -128;
    // We can only move the mouse by up to +127 units or -128 units at a time
    while ((int)currentX != (int)newX || (int)currentY != (int)newY) {
        float stepValueX;
        if (currentX == newX) {
            stepValueX = 0;
        } else if ((maxStepSizeX >= 0 && currentX+maxStepSizeX*speed > newX) || (maxStepSizeX < 0 && currentX+maxStepSizeX*speed < newX)) {
            stepValueX = fmod(newX-initialX, maxStepSizeX*speed) / speed;
        } else {
            stepValueX = maxStepSizeX;
        }
        currentX += stepValueX*speed;

        float stepValueY;
        if (currentY == newY) {
            stepValueY = 0;
        } else if ((maxStepSizeY >= 0 && currentY+maxStepSizeY*speed > newY) || (maxStepSizeY < 0 && currentY+maxStepSizeY*speed < newY)) {
            stepValueY = fmod(newY-initialY, maxStepSizeY*speed) / speed;
        } else {
            stepValueY = maxStepSizeY;
        }
        currentY += stepValueY*speed;
        bleMouse.move(stepValueX, stepValueY);
    }
}

This should be able to compensate for speed. I have no clue how to compensate for acceleration though. I don't even know how that is implemented on the OS side. It might even be implemented differently on different operating systems. Some may for example reset the acceleration to 0 once the cursor hasn't been moved for more than 2 milliseconds, others may give it 3 milliseconds and then some may slowly decrease the acceleration instead of doing it instantly... I don't know, it's very complicated and not like something that I want to deal with. Personally I disable cursor acceleration on all of my computers anyway.

Maybe this would be of help to anyone who actually wants to implement this: http://archive.is/20120907165307/msdn.microsoft.com/en-us/windows/hardware/gg463319.aspx

T-vK commented 3 years ago

Have fun adjusting this to your needs:

/**
 * Absolute mouse example
 */
#include <BleMouse.h>
#include <math.h> // required because of fmod

BleMouse bleMouse;

// Acceleration is not implemented
void moveTo(float newX, float newY, float speed = 1, float acceleration = 0, bool isLandscape = true, float initialX = 0, float initialY = 0) {
    if (!isLandscape) { // swap x and y in portrait mode
        float tmp;
        tmp = newX;
        newX = newY;
        newY = tmp;
        tmp = initialX;
        initialX = initialY;
        initialY = tmp;
    }

    float currentX = initialX;
    float currentY = initialY;

    float maxStepSizeX = (initialX < newX) ? 127 : -128;
    float maxStepSizeY = (initialY < newY) ? 127 : -128;
    // we can only move the mouse by up to +127 units or -128 units at a time
    while ((int)currentX != (int)newX || (int)currentY != (int)newY) {
        float stepValueX;
        if (currentX == newX) {
            stepValueX = 0;
        } else if ((maxStepSizeX >= 0 && currentX+maxStepSizeX*speed > newX) || (maxStepSizeX < 0 && currentX+maxStepSizeX*speed < newX)) {
            stepValueX = fmod(newX-initialX, maxStepSizeX*speed) / speed;
        } else {
            stepValueX = maxStepSizeX;
        }
        currentX += stepValueX*speed;

        float stepValueY;
        if (currentY == newY) {
            stepValueY = 0;
        } else if ((maxStepSizeY >= 0 && currentY+maxStepSizeY*speed > newY) || (maxStepSizeY < 0 && currentY+maxStepSizeY*speed < newY)) {
            stepValueY = fmod(newY-initialY, maxStepSizeY*speed) / speed;
        } else {
            stepValueY = maxStepSizeY;
        }
        currentY += stepValueY*speed;     
        bleMouse.move((char)stepValueX, (char)stepValueY);
        Serial.print((int)stepValueX);
        Serial.print("x");
        Serial.println((int)stepValueY);
        Serial.println("Moved!");
    }
}

// Only works if the target device has a cursor speed of 1x and no acceleration
// and resets the cursor position to 0|0 before every move.
void simpleMoveTo(float x, float y) {
    moveTo(0,0,1,0,true,10000,10000); // Move mouse to position 0|0
    moveTo(x,x,1,0,true,0,0); // Move mouse to position x|y
}

void setup() {
  Serial.begin(115200);
  Serial.println("Starting BLE work!");
  bleMouse.begin();
}

void loop() {
  if(bleMouse.isConnected()) {

    // Example 1:
    int screenWidth = 1920-1; // -1 because the first pixel is 0
    int screenHeight = 1080-1; // -1 because the first pixel is 0
    int cursorSpeed = 1;
    int cursorAcceleration = 0;
    int landscape = true;
    int newX = 0;
    int newY = 0;
    int oldX = screenWidth;
    int oldY = screenHeight;
    Serial.println("Moving mouse to the top left corner");
    moveTo(newX,newY,cursorSpeed,cursorAcceleration,landscape,oldX,oldY);
    delay(1000);
    Serial.println("Moving mouse to the bottom right corner");
    newX = screenWidth;
    newY = screenHeight;
    oldX = 0;
    oldY = 0;
    moveTo(newX,newY,cursorSpeed,cursorAcceleration,landscape,oldX,oldY);
    delay(1000);

    // Example 2 (doesn't require screen resolution and assumes a mouse speed of 1x):
    simpleMoveTo(500,500); // Move cursor to position 500|500
    simpleMoveTo(1000,10); // Move cursor to position 1000|10

  }
}

But as I said it does not compensate for acceleration!

xcarcelle commented 3 years ago

@T-vK : thanks a lot for this absolute example and I have been testing it with Android 8.0 on Huawey MediaPad T5. Below some adjustments on your code to make it work :

negative movement down to -127 (not -128 which overflows the range)
simpleMoveTo was calling x twice
on my case I need a 1.5 correction factor between the target coordinates (x, y) and the actual position on the tablet screen. Speed-wise it is quite fast and I will run some more tests to see if there is some glitches on the final movement. Accuracy is quite good as you use float for coordinates. Available to keep adjusting this example and make tests on more devices. Cheers.
```
/**
* Absolute mouse example
*/
#include <BleMouse.h>
#include <math.h> // required because of fmod
```

BleMouse bleMouse;

// Acceleration is not implemented void moveTo(float newX, float newY, float speed = 1, float acceleration = 0, bool isLandscape = true, float initialX = 0, float initialY = 0) { if (!isLandscape) { // swap x and y in portrait mode float tmp; tmp = newX; newX = newY; newY = tmp; tmp = initialX; initialX = initialY; initialY = tmp; }

float currentX = initialX;
float currentY = initialY;

float maxStepSizeX = (initialX < newX) ? 127 : -127;
float maxStepSizeY = (initialY < newY) ? 127 : -127;
// we can only move the mouse by up to +127 units or -128 units at a time
while ((int)currentX != (int)newX || (int)currentY != (int)newY) {
    float stepValueX;
    if (currentX == newX) {
        stepValueX = 0;
    } else if ((maxStepSizeX >= 0 && currentX+maxStepSizeX*speed > newX) || (maxStepSizeX < 0 && currentX+maxStepSizeX*speed < newX)) {
        stepValueX = fmod(newX-initialX, maxStepSizeX*speed) / speed;
    } else {
        stepValueX = maxStepSizeX;
    }
    currentX += stepValueX*speed;

    float stepValueY;
    if (currentY == newY) {
        stepValueY = 0;
    } else if ((maxStepSizeY >= 0 && currentY+maxStepSizeY*speed > newY) || (maxStepSizeY < 0 && currentY+maxStepSizeY*speed < newY)) {
        stepValueY = fmod(newY-initialY, maxStepSizeY*speed) / speed;
    } else {
        stepValueY = maxStepSizeY;
    }
    currentY += stepValueY*speed;     
    bleMouse.move((char)stepValueX, (char)stepValueY);
    Serial.print((int)stepValueX);
    Serial.print("x");
    Serial.println((int)stepValueY);
    Serial.println("Moved!");
}

}

// Only works if the target device has a cursor speed of 1x and no acceleration // and resets the cursor position to 0|0 before every move. void simpleMoveTo(float x, float y) { moveTo(0,0,1,0,true,10000,10000); // Move mouse to position 0|0 moveTo(x,y,1,0,true,0,0); // Move mouse to position x|y }

void setup() { Serial.begin(115200); Serial.println("Starting BLE work!"); bleMouse.begin(); }

void loop() { if(bleMouse.isConnected()) {

// Example 1:
int screenWidth = 1920-1; // -1 because the first pixel is 0
int screenHeight = 1080-1; // -1 because the first pixel is 0
int cursorSpeed = 1;
int cursorAcceleration = 0;
int landscape = true;
int newX = 0;
int newY = 0;
int oldX = screenWidth;
int oldY = screenHeight;
Serial.println("Moving mouse to the top left corner");
moveTo(newX,newY,cursorSpeed,cursorAcceleration,landscape,oldX,oldY);
moveTo(0,0,1,0,true, 10000, 10000); // Move mouse to position 0|0
delay(1000);
Serial.println("Moving mouse to the bottom right corner");
newX = screenWidth;
newY = screenHeight;
oldX = 0;
oldY = 0;
moveTo(newX,newY,cursorSpeed,cursorAcceleration,landscape,oldX,oldY);
delay(1000);

// Example 2 (doesn't require screen resolution and assumes a mouse speed of 1x):
//simpleMoveTo(500,500); // Move cursor to position 500|500
//delay(1000);
//simpleMoveTo(1000,10); // Move cursor to position 1000|10
//delay(1000);
simpleMoveTo(round(100/1.5),round(100/1.5));
delay(1000);
simpleMoveTo(round(522/1.5),round(421/1.5));
delay(1000);
simpleMoveTo(round(431/1.5),round(150/1.5));
delay(1000);

} }