search

w3c / mediacapture-screen-share-extensions

Other
1 stars 0 forks source link

Exposing mouse events from captured surfaces #2

Open eladalon1983 opened 1 year ago

eladalon1983 commented 1 year ago

Web applications can observe mouse events occurring over themselves. But what about events over a captured display surface? Mouse coordinates can be read by inspecting frames and searching for the cursor, but that is neither simple, nor efficient, nor robust.

During the first Screen Capture Community Group meeting (slides, minutes), @ldenoue and I described the use-cases this would unlock and presented a proposed API. The proposal was met with enthusiasm, with unanimous agreement (via poll) for the presenters to produce a draft and present it to the group during the next meeting. This draft is now ready: https://screen-share.github.io/captured-mouse-events/)

I'd like to encourage everyone interested to join the discussion. Comments on this thread, issues on the proposal's repo, participation in the next SCCG meeting, courier pigeon notes - you name it.

(CC @happylinks)

happylinks commented 1 year ago

Thanks for kicking this off @eladalon1983! Happy to get this proposal off the ground :) Added some questions to the github repo to get the discussion started.

jozefchutka commented 1 year ago

It seems I have just proposed very same thing https://github.com/w3c/mediacapture-screen-share/issues/267 ... closing my issue now as it is being in progress here

youennf commented 1 year ago

Looking quickly at the proposal, there might be some potential sync issues between video frames and the mouse events. Hopefully, this should not be a very big issue. Was it discussed though?

An alternative is to attach the mouse information as metadata to VideoFrames directly. It would be exposed to JS either via VideoFrameCallbackMetadata or via VideoFrameMetadata.

Has this option been investigated?

eladalon1983 commented 1 year ago

Has this option been investigated?

There's an issue for roughly that question. There were a few discussions on the topic, but they've not all been captured on GitHub. Attaching mouse metadata is indeed one option that's come up. Another option is to ensure there are good timestamps from the same clock source on both VideoFrames and the new events, so that the relative order can be accurately determined. And François also mentioned RVFC. You're very welcome to contribute to that discussion.