Share mouse cursor

[mattkrick]

When i'm in a meeting, i want to share my mouse cursor with other folks whenever i hold down the shift key.

There are a few challenges with this:

• My screen coordinates aren't your screen coordinates
• My scroll position isn't your scroll position

solving for scroll position would require sharing the element that i'm on top of and the scrollTop of that element. Then, if that isn't visible for another client, we just show some down arrows like super smash bros. if you click those down arrows, it'll auto scroll to where i am.

the scroll coordinates problem is harder. imagine i'm in portrait & you're in landscape. it's not enough for you to know the %x and %y of the viewport that i'm at because you really want to know what i'm on top of. conversely, if i share the element that i'm on top of & i move 2px off of it, i should still be near that thing.for example, if i'm in between cards 2 & 3, my mouse is currently in the body, but i want to share that i'm between cards 2 and 3. my 2 & 3 might be LTR, and your 2 & 3 might be top to bottom. when i drag my mouse from the left of 2 to the right of 3, how can i transpose that to the top of 2 to the bottom of 3 for you? that's not possible, so we need to compromise somewhere. Priority number 1 is to put the cursor on top of the thing that i'm on top of.

SPEC - SERVER

Each mousemove sends the following data:

className: '123abc'
ancestorsToClimb: 5,
ancestorKey: '5442235',
elWidth,
elHeight,
elX,
elY,
x,
y,
scrollTop

1. className is gathered from the mousemove event target element. if the element does not have a class, it climbs recursively until it finds one. This becomes the reference node.
2. Once a reference node is found, we call node.__reactInternalIntersance$... and recursively look at the return prop until we find a nonnull key value or the absence of another return prop. Record how many times we looked into return and the key at that time. That becomes ancestorsToClimb, ancestorKey.
3. call node.getBoundingClientRect() to get the left, top, width, height of the reference node.
4. Get x,y from the mousemove event itself
5. The elX, elY coords tell you where within the element the cursor is: Math.abs(x - left), Math.abs(y - top).

SPEC - CLIENT

1. call document.getElementsByClassName(className) to get a list of the possible elements
2. for each element, safely look into the return property ancenstorsToClimb times
3. if the ancestorKey matches (it could be null), break. if it's never found, return & ignore the payload
4. call node.getBoundingClientRect() to get the left, top, width, height of the reference node.
5. If the size & aspect ratio are similar (heuristic TBD), determine coordinates by elX / elWidth, elY / elHeight. Memoize & record event in a stack
6. If the size & aspect ratio are not similar, determine a movement diff based on the previous events (xDiff = x(t) - x(t-1)). if the cursor is within NEARBY_THRESHOLD (TBD), add the diff to the previous event to determine the location. Multiply it by an element area coefficient: myElementWidth / serverElementWidth) * xDiff + x(t-1). For example, if the server was 5 pixels away from the right border of a button & the next event shows 5 pixels to the right of that button & my screen is 90% the width of the server, this will find a diff of 10 * .9 = 9 pixels to the right of the previous location.
7. If there is no previous event inside a fixed component, or the NEARBY_THRESHOLD is exceeded, use the naive estimation shown in Step 5.

@jordanh we're basically building a coordinate transposition engine here. any holes you can poke in this now will save a ton of time later!

---

use cases:

• when i hold shift & jiggle my mouse, you should see it jiggle on your screen over the same element i want attention brought to
• if my jiggling gets a little crazy & i go past the borders of the intended component, or i circle the intended component with my cursor, you should see that as well

assumptions:

• our smaller components, like cards, buttons, menus, etc. do not change aspect ratio
• our smaller components do not change much in size
• our larger components can change a bunch in both size & aspect ratio

shortcomings:

• being near an element without ever being inside it, for example circling a card without first entering the card, will not translate well
• generally, any element that varies dramatically in size, such as large containers that house other smaller element, will not perform well

---

Options considered to arrive at the spec:

• do nothing. your raw x, y become my x, y. this only works if we have the exact some screen coords
• transpose your x,y, innerHeight, innerWidth to a %x %y and use that with my innerHeight, innerWidth. this only works if every component on the page is directly proportional. Since a 10px button doesn't shrink to 8 pixels when i decrease width from 1000px to 800px, this won't show me exactly what you're hovering over
• transpose your %xComponent,%yComponent ,hoveredComponent directly to mine. in other words, if you are 50% left and 50% down on the button component, i know exactly where to put my cursor. the problem is if you're not on the button component, but the background nearby it, since that shape will vary widely
• for every x, y, that you send, before sending it, see if you're in a "large" area. if so, see if there is any component within a certain threshold of your cursor (e.g. 8 pixels) and then send something like "6 pixels from the bottom of the button"
• a cheaper alternative would be keeping a list of components i was just on. then if the underlying component changes, i can diff the x,y so i know that i moved 10 pixels up and i'm not 5 pixels above the button i was just on
• when i receive the component that you're in, i measure that component & see if the size is "large" and see if it's roughly the same dimensions as mine. if aspect ratio is about the same, a % if fine to use. if your thing is really wide mine is narrow (eg you're on the background of your viewport and i'm on my phone) then i have to fallback to the above "what's-nearby" approximation. Unfortunately, it's really, really hard to tell someone what component you're hovering over. DOM trees are not identical (eg if i'm facilitator, i have the facilitator bar, etc). Nodes don't have unique IDs that persist across devices, etc.

---

effort: 13 points

acceptance criteria:

• User A is in the group phase and the window is wide enough for 4 columns

• User B is in the group phase and window is wide enough for 3 columns

• [ ] User A: hold down the mouse cursor when in between card 1 and 2, it appears for User B

• [ ] User A: move cursor to card 1, the movement is mirrored for User B

• [ ] User A: draw a circle around card 1, the movement is mirrored for User B

• [ ] User A: leave card 1 and without touching any other card, go below card 4. the cursor is under card 3 for User B

• [ ] User A: move into card 4, the cursor jumps to card 4 for User B

• [ ] scrolling behavior & alerts are NOT handled in this PR

[mattkrick]

i think we arrived at a good heuristic in the grouping phase. we can use that same heuristic for other areas where we want to share cursor position

[jordanh]

I still think this is a good idea. Could we re-open this one @mattkrick ?

[mattkrick]

why sure!

[github-actions[bot]]

Stale issue

[mattkrick]

shuttin er down