No keyboard or mouse

[m7mdcc]

Hello , Thanks for your work .. i test your server example but no keyboard or mouse is working .. is that normal ? is there any option to enable this? or is it default to be viewonly ? any option to accept remote input ?

Thanks

[qmfrederik]

Hi @m7mdcc ,

Keyboard and mouse input has been implemented in the remote viewing library (at least the base infrastructure is here), but so far I've focussed on read-only viewing of a remote screen so I can't tell you (yet) whether it's fully functional :-).

I'll probably get round to validating that scenario in the coming months, but in meanwhile, feel free to send a PR should you get it working :-)

[ljruiten]

@qmfrederik I am willing to get started on that PR. But the major part of the work will be in mapping X11 keysyms to something that windows will understand through something like the SendKeys function.

This is also not a trivial task to do, because for as far as I know Windows does not have a symbol set that directly maps to all the possibilities in the X11 Key Symbols.

For my solution I will probably start by taking Windows' virtual keyset and start translating those to X11 symbols and add something like a VncVirtualKeyboard class. Is this something you can agree with?

[qmfrederik]

Hi @ljruiten ,

Sounds good, thanks! I'd guess that if you start with implementing the ASCII keys and the common key simplemented on page 23 in the VNC Protocol Specification, we'd already get a long way.

You should be able to use the raw SendInput method via P/Invoke or SendKeys.SendWait, I have a slight preference for using P/Invoke as I'm trying to reduce the dependency on System.Drawing (mainly because .NET Core doesn't include System.Drawing), but we can always refactor later.

Let me know if you start working on this, looking forward to the PR!

[ljruiten]

Started already, just forgot to start working in my own fork... (doink)

The VirtualKeyboard is going quite well, I already got a proof of concept working with SendInput for a subset of the ASCII keys (the alphabet characters). The other keys will just be a bit of tedious copy pasting. How do you think I should handle unimplemented X11 keys? Just drop and ignore them?

The VirtualMouse is not as trivial as the keyboard, the PointerChangedEventArgs contain the coordinates and the state of each button. This is not how sendinput works tough, it works with mousekeydown and mousekeyup events. So the VirtualMouse will need to remember it's state by with the help of some booleans I guess? And compare that to the new state each time the event is fired?

[qmfrederik]

If you think you're making good progress on VirtualKeyboard, you may want to consider creating that on a seperate branch and already sending a PR for that :)

Regarding the VirtualMouse, you are right that SendInput works with high-level mouse concepts (such as click,...) and you would need to keep track of the mouse state yourself. Did you look at mouse_event (p-invoke definitions)? It seems that this function accepts lower-level mouse input so it may be a better match.

I think you can silently drop input that is not recognized - like unknown keyboard events. Another option would to log it using Debug.WriteLine or using LibLog to implement logging.

[ljruiten]

Ah you right. Yes I looked at it, but appearently not long enough. It does indeed accept the state of the buttons as opposed to the changing of the state.

I am unfamiliar with the styling monitor that is used in this project. I get style erros for almost every single line of code in the project (including the code that was already there). Is this normal or am I doing something wrong?

[qmfrederik]

We're using a subset of StyleCop Analyzers to enforce a common style.

I'm not seeing any StyleCop warnings in a clean checkout from git. It may be because some of your Visual Studio settings (tabs vs spaces, just to name one) conflict with the StyleCop settings.

There are code fixes available for most StyleCop errors -- e.g. you either click the lightbulb or hit SHIFT+F10 and there should be an option to automatically fix most of the issues it finds.

You should be able to (temporarily) remove the StyleCop.Analyzers NuGet package to suppress the warnings, but I do expect PR's to adhere to the StyleCop guidelines.

[ljruiten]

Well you must be using some suppress warnings that are not part of the project. I am getting the SA1200 a using directive must appear withing a namespace decleration warning. I get this on a new clone from git and at home at my basically fresh install of VS2015 too.

[ljruiten]

Sorry for not actively working on this! Some other work got assigned to me, but I will continue with this in a month or so...

[johnnyontheweb]

@qmfrederik is there any news on that? As far as I read, does the support for mouse and keyboard is to be implemented? I noticed that it is missing also on server side. Can you be more precise where to place code to try to implement keyb. and mouse support? thanks

[qmfrederik]

@johnnyontheweb Not much news at the moment, sorry.

@ljruiten did some work on that which I think is still on his branch: https://github.com/ljruiten/remoteviewing/commits/master - have a look at the last two commits. From looking at the code, it does seems like he managed to implement a virtual mouse and keyboard and hook it up, but I'm not sure how functional it is at the moment.

@ljruiten If you have anything which you think is ready to be shared, even if only partial, I take pull requests :)

[ljruiten]

Yes about that... I have actually done some work on this today and will commit it to my fork tomorrow. For the keyboard part I have now implemented most of the ASCII keys, and it seems to work. However currently not all keys are implemented, but that is just a matter of going through the list of X11 key definitions combining them with the ASCII key list.

The mouse part is what I tried to get working today, and I could get some clicks through. The way VNC sends the mouse state is not directly compatible with how the Windows API accepts it, so the mouse is a little harder to implement. I will try to add the cursor position support (which should be easy) and scrolling (less easy) tomorrow and commit it to my fork.

[ljruiten]

@qmfrederik , @johnnyontheweb I did not get to spend much time on it today, but you can check a branch called "mouse" in my fork to see my initial implementation. It now only has left mouse clicking. The other buttons are easily added, but I am having problems with the Windows API.

As mentioned earlier in this conversation, mouse_event from user32.dll works by sending changes in the state of the button rather than continuously sending that whether the button is pressed on each update, so I added a property that keeps track of the state so that it can detect changes on each update. This works, but I currently cannot get the pointer to click at the right position. If I add the MOUSEEVENTF_MOVE it just clicks in the upper right corner, but the clicks do come through.

Update:

Fixed the cursor position issue by using SetCursorPos instead of mouse_event.

[ljruiten]

@johnnyontheweb You can now find initial mouse support in the master branch. I will add keyboard support in the next week or so. I would appreciate some more testing, so please try it out.

[johnnyontheweb]

@ljruiten thank you so much, I'll be away from office, I'll try it out asap and let you know

[johnnyontheweb]

Sorry for the long delay... I'm ready to test the app, but I noticed that I cannot do that in Windows XP since .NET 4.5 is required. I cannot change the destination framework in VS 2017, please advice on that.

I performed the test with a Win7 virtual machine: mouse OK for a few seconds, then I cannot see thw curson on the screen and the applicartion freezes. Keyboard is not working. No error is reported on server side.