search

any1 / neatvnc

A liberally licensed VNC server library with a clean interface
ISC License
118 stars 29 forks source link

Lossy hardware-accelerated video encoding #38

Closed emersion closed 2 years ago

emersion commented 4 years ago

Use-case: remote gaming. Connect to a beefy machine on the LAN, start an OpenGL/Vulkan game and play remotely.

Right now it's not really possible to play remotely like this with WayVNC. Screen updates are too slow and cursor lags behind. I think hardware-accelerated video encoding (e.g. VA-API) could help.

Probably needs a new VNC extension.

jadahl commented 4 years ago

Probably needs a new VNC extension.

VNC (or RFB I suppose) already has a H.264 extension, so that should be handled already I think. I'm not aware if there are any (free software) server or client side implementations of it yet though.

There seems also to be some concerns whether it's suitable as well, see https://turbovnc.org/About/H264.

emersion commented 4 years ago

Note, the study was made with a software H.264 encoder. A hardware encoder may help.

progandy commented 4 years ago

Two different VA-H264 encoding types already exist (number 23 and 0x48323634) as well as plain H.264 as number 20, but there is no specification of the wire protocol that I can find.

https://www.iana.org/assignments/rfb/rfb.xhtml#rfb-4

The implementation that originated 0x48323634 is probably this proposal for qemu and gtk-vnc https://lists.gnu.org/archive/html/qemu-devel/2013-02/msg01835.html https://mail.gnome.org/archives/gtk-vnc-list/2013-February/msg00000.html

Encoding 20 and 23 seem to be assigned to RealVNC.

any1 commented 4 years ago

Two different VA-H264 encoding types already exist (number 23 and 0x48323634) as well as plain H.264 as number 20, but there is no specification of the wire protocol that I can find.

I've investigated both those encodings. 0x48323634 stalled because Intel didn't create a proper open specification for it and the projects to which patches were submitted didn't like that. I contacted Tristan Richardson via email about 23 and he said that it had not been implemented by RealVNC and was not being pursued at the moment. He advised me that I should create a new extension if I wanted to implement H.264 encoding.

jadahl commented 4 years ago

Could just as well consider using another video codec in that case, as clients would need to explicitly add support for the new extension anyway.

any1 commented 4 years ago

Could just as well consider using another video codec in that case, as clients would need to explicitly add support for the new extension anyway.

Well, we could actually support multiple video codecs and negotiate with the client about which one to use. The "best" codec which is supported in hardware by both would be chosen.

emersion commented 3 years ago

Ref https://github.com/rfbproto/rfbproto/pull/39