VNC Support

[lars18th]

Hi,

I feel that this project will be more interesting if we can use the Android device as a PROXY for a physical hardware (like KVMIP server).

Here my proposal: Add VNC server tool.

To achieve this fast, in the project of UML (User Mode Linux), you can found a very simple VNC server for a virtual shared memory framebuffer. In this project the server isn't running with real vga, keyboard, mouse.

The source code is on this page: http://intgat.tigress.co.uk/rmy/shmfb/index.html If you download the source (uml_fb-0.1.1.tar.gz), inside you have the "uml_vncfb.c" tool. This tool uses the well known LibVNCServer, and it's very lightweight. In the source you have isolated the function that sends the events of VGA, KEYBOARD and MOUSE. See this on the source:

• shmfb_command(SHMFB_KEY_EVENT, keymap[i].keycode, down, 0, 0);
• hmfb_command(SHMFB_BUTTON_EVENT, vnc_mousemap[i],((now >> i) & 1), x, y);
• shmfb_command(SHMFB_UPDATE_EVENT, 0, 0, 0, 0)

So, if you replace these three calls with the code of hid-gadget-test utility you can implement a simple VNC server for this.

At time, the framebuffer can be a black window, but if you implement the camera capture, then you can implement the full remote functionality.

Someone likes to implement this?

[pelya]

I have thought about it, remote functionality is even mentioned in the plans in readme, but alas, I've got too little time to implement that. I don't think VNC protocol will do any acceptable compression on camera feed, I was going to make a separate client/server with ffmpeg or something. VNC seems easier though, because I don't need to implement client. The only issue is that client mouse cursor will be disconnected from the remote cursor, because there's no way to determine it's position without doing image recognition on camera feed.

On Thu, Jan 15, 2015 at 3:27 PM, lars18th notifications@github.com wrote:

Hi,

I feel that this project will be more interesting if we can use the Android device as a PROXY for a physical hardware (like KVMIP server).

Here my proposal: Add VNC server tool.

To achieve this fast, in the project of UML (User Mode Linux), you can found a very simple VNC server for a virtual shared memory framebuffer. In this project the server isn't running with real vga, keyboard, mouse.

The source code is on this page: http://intgat.tigress.co.uk/rmy/shmfb/index.html If you download the source (uml_fb-0.1.1.tar.gz), inside you have the "uml_vncfb.c" tool. This tool uses the well known LibVNCServer, and it's very lightweight. In the source you have isolated the function that sends the events of VGA, KEYBOARD and MOUSE. See this on the source:

• shmfb_command(SHMFB_KEY_EVENT, keymap[i].keycode, down, 0, 0);
• hmfb_command(SHMFB_BUTTON_EVENT, vnc_mousemap[i],((now >> i) & 1), x, y);
• shmfb_command(SHMFB_UPDATE_EVENT, 0, 0, 0, 0)

So, if you replace these three calls with the code of hid-gadget-test utility you can implement a simple VNC server for this.

At time, the framebuffer can be a black window, but if you implement the camera capture, then you can implement the full remote functionality.

Someone likes to implement this?

— Reply to this email directly or view it on GitHub https://github.com/pelya/android-keyboard-gadget/issues/20.

[lars18th]

Hi,

I feel that VNC is the best option for several reasons:

1) It's a common protocol and a lot of clients exists.

2) This protocol focuses on KEYBOARD, MOUSE and VIDEO. The three targets of this project.

3) The VNC can support different compression algorithms, and some versions include MPEG video (also based on ffmpeg). Don't worry about this. Standard VNC video codecs focus on SCREEN, not real video, but the users like to point the camera to the screen!

4) Modern KVM-IP devices supports VNC or are native based on VNC. The aim of this project is replicate this functionality.

5) Using the source code pointed it's very easy to implement a testing-vnc-server that sends black video, but supports KEYBOARD and MOUSE. At time, this project is useless for remote use. With this simple "example" it's possible to use any VNC client to simulate a USB keyboard and mouse connected to any device. In this case only de video is missing. And this will be a huge improvement over the current status of the project, at the cost of some few code lines.

Why not try it?

[pelya]

Okay, okay, but I'll likely use tightvncserver sources, because I need PointerPos extension, also compression provided by it would be nice.

On Fri, Jan 16, 2015 at 10:11 AM, lars18th notifications@github.com wrote:

Hi,

I feel that VNC is the best option for several reasons:

1) It's a common protocol and a lot of clients exists.

2) This protocol focuses on KEYBOARD, MOUSE and VIDEO. The three targets of this project.

3) The VNC can support different compression algorithms, and some versions include MPEG video (also based on ffmpeg). Don't worry about this. Standard VNC video codecs focus on SCREEN, not real video, but the users like to point the camera to the screen!

4) Modern KVM-IP devices supports VNC or are native based on VNC. The aim of this project is replicate this functionality.

5) Using the source code pointed it's very easy to implement a testing-vnc-server that sends black video, but supports KEYBOARD and MOUSE. At time, this project is useless for remote use. With this simple "example" it's possible to use any VNC client to simulate a USB keyboard and mouse connected to any device. In this case only de video is missing. And this will be a huge improvement over the current status of the project, at the cost of some few code lines.

Why not try it?

— Reply to this email directly or view it on GitHub https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-70220759 .

[lars18th]

Hi,

I hope you can prepare a simple test demo of VNCServer. The reason is this other project: http://danman.eu/blog/reverse-engineering-lenkeng-hdmi-over-ip-extender/

This is my idea:

. COMPUTER ---HDMI------ (Video+Audio) --------> HDMI-2-IP ----> Android device ---> VNC . . . . |-----USB--------------------------- (Keyboard+Mouse) --------------/

The HDMI-2-IP is really cheap (less than $/€100) and can send the HDMI compressed to the network in mulcast. Then any network device can capture the video (plus audio!) at ~15fps.

So, my suggestion is that your VNC server will be MODULAR to inject any video input using pipes (FIFO files). Perhaps you can read directly JPEG, or use VLC (or any other processor) to input raw data in you memory framebuffer.

If you feel that one Android device don't have sufficient power to achieve all these tasks, then my suggestion is to support BLACK video in the VNC Server. Then you can use a VNC PROXY to inject the video, as this example:

Keyboard+Mouse USB ---> Android VNC Server ---> VNC Proxy ---> Client

And the VNC Proxy integrates the JPEG-IP video stream inside the VNC connection. The proxy can be based on: http://github.com/pmdumuid/rfbproxy

You like this (crazy) idea?

[pelya]

Nah, $70 is not "really cheap", it is "not expensive". Raspberry PI type A at $25 is really cheap. PI type B and Chromecast at $35 are just cheap.

Also setting up two devices instead of just one is way too complicated for typical mobile users, I want to keep it simple, so I'll use camera.

Don't expect test demo anytime soon. Maybe in a month or two, also I'm no fan of test demos, I intend to make it usable out of the box.

On Tue, Jan 20, 2015 at 1:51 PM, lars18th notifications@github.com wrote:

Hi,

I hope you can prepare a simple test demo of VNCServer. The reason is this other project: http://danman.eu/blog/reverse-engineering-lenkeng-hdmi-over-ip-extender/

This is my idea:

COMPUTER ---HDMI------ (Video+Audio) --------> HDMI-2-IP ----> Android device ---> VNC |-----USB--------------------------- (Keyboard+Mouse) --------------/

The HDMI-2-IP is really cheap (less than $/€100) and can send the HDMI compressed to the network in mulcast. Then any network device can capture the video (plus audio!) at ~15fps.

So, my suggestion is that your VNC server will be MODULAR to inject any video input using pipes (FIFO files). Perhaps you can read directly JPEG, or use VLC (or any other processor) to input raw data in you memory framebuffer.

If you feel that one Android device don't have sufficient power to achieve all this tasks, my suggestion is to support BLACK video in the VNC Server. Then you can use a VNC PROXY to inject the video, as this example:

Keyboard+Mouse USB ---> Android VNC Server ---> VNC Proxy ---> Client

And the VNC Proxy integrates the JPEG-IP video stream inside the VNC connection.

You like this (crazy) idea?

— Reply to this email directly or view it on GitHub https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-70642656 .

[lars18th]

Yes, you're right!

However, a very simple vnc server (with black screen, it's say no-video) implemented using a simple library (perhaps based on the code I was commented) can be very useful. Please, think you on a remote software, running on another device, that likes to send Keyboard and/or Mouse commands over the USB. With this simple vnc server it's posible to use any Android device as a TUNNEL for send these commands. You don't need to create any new protocol to send this info. And the API it's the simple RFB protocol.

I hope you think on create a "test" version based on the simple "hid-gadget-test.c", and send command using shmfb_command() function present in "uml_vncfb.c".

But you can be confident, I will have patience. ;-)

[alandewayne]

Stop sending me emails On Jan 21, 2015 1:35 AM, "lars18th" notifications@github.com wrote:

Yes, you're right!

However, a very simple vnc server (with black screen, it's say no-video) implemented using a simple library (perhaps based on the code I was commented) can be very useful. Please, think you on a remote software, running on another device, that likes to send Keyboard and/or Mouse commands over the USB. With this simple vnc server it's posible to use any Android device as a TUNNEL for send these commands. You don't need to create any new protocol to send this info. And the API it's the simple RFB protocol.

I hope you think on create a "test" version based on the simple "hid-gadget-test.c", and send command using shmfb_command() function present in "uml_vncfb.c".

But you can be confident, I will have patience. ;-)

— Reply to this email directly or view it on GitHub https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-70801541 .

[alandewayne]

STOP sending me emails On Jan 16, 2015 9:46 AM, "Sergii Pylypenko" notifications@github.com wrote:

Okay, okay, but I'll likely use tightvncserver sources, because I need PointerPos extension, also compression provided by it would be nice.

On Fri, Jan 16, 2015 at 10:11 AM, lars18th notifications@github.com wrote:

Hi,

I feel that VNC is the best option for several reasons:

1) It's a common protocol and a lot of clients exists.

2) This protocol focuses on KEYBOARD, MOUSE and VIDEO. The three targets of this project.

3) The VNC can support different compression algorithms, and some versions include MPEG video (also based on ffmpeg). Don't worry about this. Standard VNC video codecs focus on SCREEN, not real video, but the users like to point the camera to the screen!

4) Modern KVM-IP devices supports VNC or are native based on VNC. The aim of this project is replicate this functionality.

5) Using the source code pointed it's very easy to implement a testing-vnc-server that sends black video, but supports KEYBOARD and MOUSE. At time, this project is useless for remote use. With this simple "example" it's possible to use any VNC client to simulate a USB keyboard and mouse connected to any device. In this case only de video is missing. And this will be a huge improvement over the current status of the project, at the cost of some few code lines.

Why not try it?

— Reply to this email directly or view it on GitHub < https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-70220759>

.

— Reply to this email directly or view it on GitHub https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-70282513 .

[lars18th]

Hi Pelya,

Related to the project of a VNC server... I try to execute this:

1) Install Xfvb and X11vnc in some Linux machine (can be your Android device or another computer).

2) Execute a virtual X11 session, for example with commands:

$ Xvfb :1 -screen 0 1024x768x24+32 & $ DISPLAY=:1 blackbox & (if you like to execute some Window Manager) $ DISPLAY=:1 x11vnc &

3) Then use any remote computer to connect to your VNC server.

4) Inside the X11 session execute command "xinput" for capture keyboard events. First list devices:

$ xinput --list $ xinput test 'Virtual core XTEST keyboard' $ xinput test 'Virtual core XTEST pointer'

Then if you capture the ouput of 'xinput test' then you can process it in the Android device. You can send it using Netcat and pipes; or you can execute xinput in Android and connect to the X11 using TCP/IP (I test to execute xinput remotely and it works). You only need the binary of xinput inside the Android device!

My idea is execute as a background of the X11 session the video capture of the remote screen. Then you don't need to implement any VCN server, only a new test-tool that can undesrtand the "xinput test" data.

What you think? Can you mix "xinput" and "hid-gadget-test" ? Regards.

PD: Some examples of outputs running xinput on remote machine:

• Output of "xinput test 'Virtual core XTEST keyboard'": key press 111 key release 111 key press 36 key release 36
• Output of "xinput test 'Virtual core XTEST pointer'": motion a[0]=641 a[1]=226 motion a[0]=641 a[1]=225 motion a[0]=646 a[1]=223

All working in realtime! No problem at all! Only need to pipe this output to your test-tool. If you like to compile (static) the xinput tool, the source code is here: http://cgit.freedesktop.org/xorg/app/xinput/tree/src You only need a TCP socket to the X11 server to run the xinput in the Android device!

[pelya]

Yes it's pissible, but kind of complicated. I'll just cross-compile tightvncclient, I know how to do that and I've done such things before, the only thing I need is two days of free time, which I currently don't have. On Jan 29, 2015 7:50 PM, "lars18th" notifications@github.com wrote:

Hi Pelya,

Related to the project of a VNC server... I try to execute this:

1) Install Xfvb and X11vnc in some Linux machine (can be your Android device or another computer).

2) Execute a virtual X11 session, for example with commands:

$ Xvfb :1 -screen 0 1024x768x24+32 & $ DISPLAY=:1 blackbox & (if you like to execute some Window Manager) $ DISPLAY=:1 x11vnc &

3) Then use any remote computer to connect to your VNC server.

4) Inside the X11 session execute command "xinput" for capture keyboard events. First list devices:

$ xinput --list $ xinput test 'Virtual core XTEST keyboard' $ xinput test 'Virtual core XTEST pointer'

Then if you capture the ouput of 'xinput test' then you can process it in the Android device. You can send it using Netcat and pipes.

My idea is execute as a background of the X11 session the video capture of the remote screen. Then you don't need to implement any VCN server, only a new test-tool that can undesrtand the "xinput test" data.

What you think? Can you mix "xinput" and "hid-gadget-test" ? Regards.

— Reply to this email directly or view it on GitHub https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-72070796 .

[lars18th]

Hi Pelya,

I hope you found free time soon. Thank you for your comments!

Only one simple point for discuss: If you cross-compile the "xinput" tool, it will be a very good improvement. Perhaps you found quite complicated my proposal, but it's really simple. If we have "xinput" compiled for Android, we can do that in the Android shell:

$ DISPLAY:remote_ip:1 xinput test 'Virtual core XTEST keyboard' | hid-gadget-test /dev/hidg0 xkeyboard & $ DISPLAY:remote_ip:1 xinput test 'Virtual core XTEST pointer' | hid-gadget-test /dev/hidg1 xmouse &

Then, all mouse/keyboard events in the remote X11 session will be processed by the "hid-gadget-test" tool. Please, note here the "xkeyboard" and "xmouse" parameters. These indicates to read standard input with xinput sintax.

Please, if you point to me about how to cross-compile the xinput tool then I'll try! Regards.

PD: At time, I'll experiment with "Linux Deploy" and a Debian running in the Android device. With this I can try the semi-native xinput tool. Next step is modify "hid-gadget-test" to understand xinput sintax (at least for keyboard events, less complex to implement that for mouse).

[lars18th]

Hi Pelya,

Let me to explain my last test:

1) I do an install of Debian (ARMhf) using Linux Deploy. I access it using ssh. Install on it the xinput tool (sudo apt-get install xinput).

2) Execute in a remote PC the Xfvb and X11vnc.

3) From the Linux shell in the Android device I execute:

$ ssh -L 6002:127.0.0.1:6001 user@remote-pc

4) Then when you access to DISPLAY:2 in the Android device you're contacting with the X11 running with the Xfvb of the remote PC. I can access to this virtual X11 server using any VNC client.

5) In the Linux shell of Android device I can execute:

$ DISPLAY=:2 xinput test 'Virtual core XTEST keyboard' $ DISPLAY=:2 xinput test 'Virtual core XTEST pointer'

And I see all keyboard and mouse events. So for now (and until we have an standalone version of xinput for Android) we can execute these commands from the Linux shell. Please, can you write in the "Compilation" section a guide for compiling the "hid-gagdet-test"?

The next step is then do the pipe between the output of "xinput" and the input of "hid-gadget-test". I hope you can found time to help me! Thank you for this very promissing tool.

PD: If you try to use the X11 VNC server running inside the Android device (Linux Deploy can execute one of this), you can't use the xinput tool. The reason is because the running VNC server is ThightVNC (Xvnc) and it don't includes the XInputExtension (you can check it with command "xdpyinfo"). So don't try to use this VNC server with xinput!

[lars18th]

Hi Pelya,

Related to my idea of implementing a VNC server in a different device, I like to explain my advances:

• If you like to remote send commands to /dev/hidgX, you can use this simple technique: Use SOCAT to create a PIPE in a remote machine that is connected to the character device. Then, in the remote machine you can execute ANY program that uses the device like it's local.

For example:

• In the shell of the Android device you can execute: "$ socat OPEN:/dev/hidg0 tcp-l:5000"
• In the shell of the remote (Linux) machine you can execute: "$ socat tcp:192.168.1.10:5000 PIPE:/tmp/hidg0"
• Then, you can execute the "hid-gadget-test" in the remote machine. As example: "$ echo f2 | hid-gadget-test /tmp/hidg0 keyboard" and this command sends a 'F2' keystroke.

Therefore, my idea is execute the SOCAT over a SSH connection to the Android device and "export" both devices: /dev/hidg0 & /dev/hidg1. Then it's possible to modify/compile/execute any tool in the remote machine. I'll try with the way of a virtual X11 session, capturing the keyboard and sending keypress to remote "/dev/hidg0". For the video, I like to execute root-window (background) video from VideoLAN or MPlayer. This video can be from a capture device or from the Android camera (several Apps for Video-2-IP exists).

If you need the native SOCAT binary for Android, you can found one here: http://forum.xda-developers.com/showpost.php?p=51837955&postcount=28

Please, comment what you think. ;-) Regards.

[pelya]

I've fixed sources of USB Keyboard app, so you can compile it on generic Linux - go to directory remote-client and type 'make', you'll need SDL 1.2, SDL_ttf and SDL_image installed on your Linux. It will still try to write to /dev/hidg0 and /dev/hidg1, it's up to you to create appropriate pipes using mkfifo, and forward them to Android using two socat commands.

On Sat, Feb 7, 2015 at 12:19 AM, lars18th notifications@github.com wrote:

Hi Pelya,

Related to my idea of implementing a VNC server in a different device, I like to explain my advances:

• If you like to remote send commands to /dev/hidgX, you can use this simple technique: Use SOCAT to create a PIPE in a remote machine that is connected to the character device. Then, in the remote machine you can execute ANY program that uses the device like it's local.

For example:

-

In the shell of the Android device you can execute: "$ socat OPEN:/dev/hidg0 tcp-l:5000"

In the shell of the remote (Linux) machine you can execute: "$ socat tcp:192.168.1.10:5000 PIPE:/tmp/hidg0"

Then, you can execute the "hid-gadget-test" in the remote machine. As example: "$ echo f2 | ./hid-gadget-test ./hidg0 keyboard" and this command sends a keystroke.

Therefore, my idea is execute the SOCAT connection over a SSH connection to the Android device and "export" both devices: /dev/hidg0 & /dev/hidg1. Then it's possible to modify/compile/execute any tool in the remote machine. I'll try with the way of a virtual X11 session, capturing the keyboard and sending keypress to remote "/dev/hidg0". For the video, I like to execute root-window (background) video from VideoLAN or MPlayer. This video can be of a capture device or from the Android camera (several Apps for Video-2-IP exists).

If you need the native SOCAT binary for Android, you can found one here: http://forum.xda-developers.com/showpost.php?p=51837955&postcount=28

Please, comment what you think. ;-) Regards.

— Reply to this email directly or view it on GitHub https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-73323393 .

[pelya]

Fixed in 1.10, camera size is hardcoded to 1280x720, maybe I'll do it cpnfigurable in the future. You don't need all that xinput, socat, and mplayer stuff, there is just one button to turn it on.

[kgara]

Could you advise where to find SDL_android.h ? Could not find this header among libsdl-dev packages.

remote-client # make
g++ -g -o usb-keyboard-client gfx.cpp vnc_server.cpp flash_kernel.cpp gui_settings.cpp vnc_keysyms.cpp input.cpp gui.cpp camera.cpp main.cpp touchpad.cpp scancodes.c -lSDL_ttf -lSDL_image `sdl-config --cflags` `sdl-config --libs` -I"/usr/lib/jvm/java-8-openjdk-amd64/include" -I"/usr/lib/jvm/java-8-openjdk-amd64/include/linux"
camera.cpp:2:25: fatal error: SDL_android.h: No such file or directory
compilation terminated.
Makefile:5: recipe for target 'usb-keyboard-client' failed
make: *** [usb-keyboard-client] Error 1

[pelya]

Here: https://github.com/pelya/commandergenius.git

On Dec 2, 2017 11:19 AM, "kgara" notifications@github.com wrote:

Could you advise where to find SDL_android.h ? Could not find this header among libsdl-dev packages.

remote-client # make g++ -g -o usb-keyboard-client gfx.cpp vnc_server.cpp flash_kernel.cpp gui_settings.cpp vnc_keysyms.cpp input.cpp gui.cpp camera.cpp main.cpp touchpad.cpp scancodes.c -lSDL_ttf -lSDL_image sdl-config --cflags sdl-config --libs -I"/usr/lib/jvm/java-8-openjdk-amd64/include" -I"/usr/lib/jvm/java-8-openjdk-amd64/include/linux" camera.cpp:2:25: fatal error: SDL_android.h: No such file or directory compilation terminated. Makefile:5: recipe for target 'usb-keyboard-client' failed make: *** [usb-keyboard-client] Error 1

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-348679602, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJewGTaBMTn_Cp-Gz8-YH4aZXC5GFe-ks5s8RYggaJpZM4DSzxs .