search

novnc / noVNC

VNC client web application
https://novnc.com
Other
11.16k stars 2.24k forks source link

Specify Multi-touch Support #944

Open hsk81 opened 6 years ago

hsk81 commented 6 years ago

At www.dizmo.com, we're developing an interface (of things), and decided to go for VNC and noVNC to enable browser support for our software, which relies as well on multi-touch support. However, noVNC does not seem to have such a feature enabled.

Hence, would it be possible to have multi-touch integrated? We would be willing to also pay the developer who integrates this feature.

I've issued a bounty of 240 DAI (a stable-coin corresponding to 240 USD) to have this issue not solved, but fully specified. Please have a look at: https://gitcoin.co/issue/novnc/noVNC/944/1047

My goal with this bounty would be to have exhaustive specification, which would allow me to open a follow-up of issue where then the actual multi-touch support is implemented. To claim the bounty, please provide a pull request with a corresponding Markdown document containing the specification.

Please see also our video gallery at https://www.dizmo.com/video-gallery/, where we display some of our use cases and where we plan to use multi-touch. In our case noVNC is running within a Docker container, hence the specification should be compatible with such a context.

NOTE: The bounty is structured as a contest with an expiration date of two weeks.

samhed commented 6 years ago

This would mean an extension of the RFB protocol, it is not trivial work.

hsk81 commented 6 years ago

Is it maybe possible to provide an estimate (in terms of developer hours) w.r.t. to this RFB protocol extension? I'm not familiar with RFB, hence the question: why would multi-touch imply such an extension?

samhed commented 6 years ago

If you mean that touch events should be passed to the desktop or application in the remote session, then we do need an extension to the protocol. The protocol only allows regular mouse or keyboard input at the moment.

No I can't give you an estimate. Estimates never work out well.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 240.0 DAI (240.0 USD @ $1.0/DAI) attached to it.

hsk81 commented 5 years ago

@samhed @CendioOssman @DirectXMan12 @kanaka I've issued a bounty of 240 DAI (a stable-coin corresponding to 240 USD) to have this issue not solved, but fully specified. Please have a look at:

My goal with this bounty would be to have exhaustive specification, which would allow me to open a follow-up of issue where then the actual multi-touch support is implemented. To claim the bounty, please provide a pull request with a corresponding Markdown document containing the specification.

Please see also our video gallery at https://www.dizmo.com/video-gallery/, where we display some of our use cases and where we plan to use multi-touch. In our case noVNC is running within a Docker container, hence the specification should be compatible with such a context.

NOTE: The bounty is structured as a contest with an expiration date of two weeks.

oogetyboogety commented 5 years ago

@hsk81 Which type of specification are you leaning towards? RFB has been extended as a specification to support multitouch inputs, pressure sensitivity, and more via RFB GII standard extension in at least two major implementations that have code associated with them, starting here: https://groups.google.com/forum/#!topic/tigervnc-devel/Zupu-o6JDNI It appears in TurboVNC you have Remote Wacom Tablet mode which uses standard messages, as well as in TigerVNC. https://github.com/TurboVNC/turbovnc/blob/2e0a0fbad308b3fb53695e472679cd9b5024d48b/unix/Xvnc/programs/Xserver/hw/vnc/init.c#L974

However, these implementations have limitations. A correct proposal in my mind would specify equations for ellipsis for each contact point and its coordinates. An application of this would then natively identify gestures, for example on a tablet you might expect two fingers to zoom but one handed phone you might expect pinch to zoom.
The same messages used in RDP could be added to the GII format as well: https://msdn.microsoft.com/en-us/library/hh554669.aspx Please let me know your feedback, thanks in advance

hsk81 commented 5 years ago

@mul1sh yes, it is still open! Please note that I'm looking for a specification only (for now at least).

hsk81 commented 5 years ago

@oogetyboogety The issue is that I'm myself not experienced enough w.r.t. multi-touch support in general. So your question about

Which type of specification are you leaning towards?

Is rather difficult to answer for me. What we have is the following software stack:

We don't have multi-touch support with noVNC, and the goal of the specification would be to clearly define a roadmap to have multi-touch implemented: This may or may not involve configuring or extending other software (like x11-xerver or tigervnc) besides noVNC.

But obviously we do not want to reinvent the wheel, and have the desired minimal functionality with as little effort as possible.

RFB has been extended as a specification to support multitouch inputs, pressure sensitivity, and more via RFB GII standard extension in at least two major implementations that have code associated with them, starting here: tigervnc-devel. It appears in TurboVNC#L974 you have Remote Wacom Tablet mode which uses standard messages, as well as in TigerVNC.

So, if we could bring in multi-touch support to noVNC using somehow the work already done for RFB (GII) and TigerVNC then great, let's do that: Here, I would like a specification (more like a roadmap), which would allow me to clearly judge if it is feasible and how long it would take to have multi-touch for noVNC.

However, these implementations have limitations. A correct proposal in my mind would specify equations for ellipsis for each contact point and its coordinates. An application of this would then natively identify gestures, for example on a tablet you might expect two fingers to zoom but one handed phone you might expect pinch to zoom. The same messages used in RDP could be added to the GII format as well: https://msdn.microsoft.com/en-us/library/hh554669.aspx

I'm not interested to create new specifications (or improve existing ones), if RFB has already such a specification in place. I would prefer an existing specification.

frankchen07 commented 5 years ago

@oogetyboogety - are you still working on this issue? Just checking in :)

hsk81 commented 5 years ago

@oogetyboogety actually, yes: please see my most recent post above.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Cancelled

The funding of 240.0 DAI (240.0 USD @ $1.0/DAI) attached to this issue has been cancelled by the bounty submitter

dcommander commented 3 years ago

I looked into the possibility of implementing this in TurboVNC, since it already implements remote Wacom tablet support using the GII RFB extension. As has been pointed out above, GII does not currently accommodate multi-touch events, and it's unclear to me how those events should be implemented on the wire. I would need to develop a better understanding of how multi-touch is implemented in X Input, but since I don't have any devices (including the various Wacom tablets at my disposal) that generate such events, I don't have any way to develop that understanding at the moment.