Confero mouse cursor on SurfacePro 3

[garyfeng]

Symptom: The Confero mouse cursor (the red dot when I turn it on in the webapp) does not show the actual position of the mouse.

Details:

Source screen size (SP3): 2160x1440

Range of mouse cursor reported on the webapp: X: [-720, 1440] Y: [480, -960]

i.e., at (0,0) on the SP3 screen, the webapp reports (-720, 480), which corresponds to about (1/3, 1/3) on the webapp video stream. At (2160, 1440), the webapp reports (1440, -960), which is outside of the video stream canvas on the webapp.

[garyfeng]

Also, the screencast video, both the streamed version and the saved version, show a weird problem with the mouse icon in the video. Specifically, on the SP3 native screen 2160x1440, when I moved the mouse to the lower right corner, the mouse icon just moved to (1440, 960), i.e., it's scaled by x0.75 in both dimensions.

Even more strange, when I used VLC to directly record the desktop via ScreenCaptureRecorder, I get a full screen video with no problem with the mouse position. The mouse can get to (2160, 1440).

I assume it's not an FFMPEG problem, because the mouse icon is added by ScreenCaptureRecorder, see https://github.com/rdp/screen-capture-recorder-to-video-windows-free/search?utf8=%E2%9C%93&q=addMouse, and line 97 of DibHelper.cpp. Doesn't see to be any issue with the scaling.

Unless, of course, somehow SCR thinks it is tracking a particular window and therefore needs to scale. Monitor m_iHwndToTrack.

[garyfeng]

A work-around at this point: to disable mouse recording in ScreenCaptureRecording. Add mouse later using the CORRECTED Confero mouse position.

This means:

• we must correctly rescale the Confero mouse events
• we can't reconstruct the icon of the mouse -- hand, arrow, resizer, etc.

But this is better than leaving false mouse trails in the videos.

[isolver]

Has the setting in app_config that defines the screen resolution in use been updated to match the S3?:

ConferoView/settings/app_config.yaml: screen_capture->screen_resolution ConferoTrack/settings/app_config.yaml: screen_capture->screen_resolution

Thanks.

On Thu, Feb 19, 2015 at 4:31 PM, garyfeng notifications@github.com wrote:

Symptom: The Confero mouse cursor (the red dot when I turn it on in the webapp) does not show the actual position of the mouse.

Details:

Source screen size (SP3): 2160x1440

Range of mouse cursor reported on the webapp: X: [-720, 1440] Y: [480, -960]

i.e., at (0,0) on the SP3 screen, the webapp reports (-720, 480), which corresponds to about (1/3, 1/3) on the webapp video stream. At (2160, 1440), the webapp reports (1440, -960), which is outside of the video stream canvas on the webapp.

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/101.

[isolver]

On Thu, Feb 19, 2015 at 11:49 PM, garyfeng notifications@github.com wrote:

Specifically, on the SP3 native screen 2160x1440, when I moved the mouse to the lower right corner, the mouse icon just moved to (1440, 960), i.e., it's scaled by x0.75 in both dimensions

This maybe an issue with the S3 video driver. On my S1, the reported screen resolution in the Windows display settings is 1920x1080, but when I get the resolution from opengl, it says the actual screen resolution is less than this. So someone is not telling the truth. ;) Perhaps something similar is happening on the S3: since ConferoView and ScreenCaptureRecorder add the mouse based on the API reported screen resolution.

You can get Track to print what it thinks the screen resolution is if you edit ConmferoTrack/track/dataCollection.py and add insert the following at line 55:

line 55 should have:

self.display = self.devices.display

#

If this is not what is on line 55 in your source, send me your copy of

the file.

otherwise, add the following after line 55

# print("Display Resolution Detected: ", self.display.getPixelResolution())

Make sure the indentation is correct. Now run Track so that you can see what is printed when it runs, when you start a recording session from View, the added print should print out what ConferoTrack has been told the screen resolution is at that point in time.

[isolver]

RE:

A work-around at this point:... Add mouse later using the CORRECTED Confero mouse position.

Unless the something /funny/ is going on with the S3 actual display resolution vs actual display resolution, there should be no need for this. Check that the screen resolution settings in the Confero app_settings.yaml files is correct.

If The S3 is like my S1, and Windows claims the screen resolution != what opengl states the screen resolution as being, then there is not much we can do about that other than do a correction as you mention.

What about Calibration Screen

If there is a fundamental issue with reported vs actual screen resolution, then you should see issues with eye data pixel positions / ranges after calibration.

As well, if you run track in a way that allows you to see the stdout (i.e. in the command prompt window), then when you run calibration, PsychoPy should spit out a warning that the requested full screen calibration screen size != native screen resolution, and that it is using native screen resolution.

[garyfeng]

Yes. Both have been set to the native resolution 2160x1440. And this has nothing to do with the Ffmpeg scaling factor. I had thought this was a bug with how sp3 reports the mouse location, but when I used VLC to record the sp3 screen directly the mouse was ok.

On Friday, February 20, 2015, Sol Simpson notifications@github.com wrote:

Has the setting in app_config that defines the screen resolution in use been updated to match the S3?:

ConferoView/settings/app_config.yaml: screen_capture->screen_resolution ConferoTrack/settings/app_config.yaml: screen_capture->screen_resolution

Thanks.

On Thu, Feb 19, 2015 at 4:31 PM, garyfeng <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Symptom: The Confero mouse cursor (the red dot when I turn it on in the webapp) does not show the actual position of the mouse.

Details:

Source screen size (SP3): 2160x1440

Range of mouse cursor reported on the webapp: X: [-720, 1440] Y: [480, -960]

i.e., at (0,0) on the SP3 screen, the webapp reports (-720, 480), which corresponds to about (1/3, 1/3) on the webapp video stream. At (2160, 1440), the webapp reports (1440, -960), which is outside of the video stream canvas on the webapp.

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/101.

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/101#issuecomment-75241587.

-- gary

[isolver]

I had thought this was a bug with how sp3 reports the mouse location, but when I used VLC to record the sp3 screen directly the mouse was ok.

How do we know that VLC is not just capturing the actual mouse cursor graphic as part of the overall screen capture?

Confero View and the Screen Capture Filter (correct?) rely on Windows API to get the mouse position, and then draw mouse cursor graphic. So does this not suggest that there is a difference somewhere between what the reported screen resolution is vs. what resolution bounds mouse position is being reported in by the S3?

[garyfeng]

Certainly true with Confero View's mouse cursor (the red dot). But for the streamed video, I thought it was set up so that FFMPEG gets frames from ScreenCaptureRecorder (a directDraw filter) with the mouse cursor drawn into the frame, see the addMouse() function at https://github.com/rdp/screen-capture-recorder-to-video-windows-free/search?utf8=%E2%9C%93&q=addmouse

There is a possibility that SP3 reports the mouse location in a different way, which causes all these weird behaviors. What I haven't done is to use FFMPEG (instead of VLC) to capture from ScreenCaptureRecorder manually, and see if the mouse is set up correctly. Will test in a few days (swamped by deadlines now).

On a different note, reducing the FFMPEG scaling factor from 1.0 to 0.1 indeed reduced the CPU by a small margin in my eye-balling (from 60+% to 50+% ish). This is when the SP3 is running both Track and View but the Webapp is running on a different computer, though I also ran into severe network throttling in the wifi when I tested this in my office (for no good reason the connection between my laptop and SP3 was <100k on the same wifi network; I have only 1 ethernet outlet in my office).

Overall FFMPEG took about 30-40% CPU, by far the heaviest process on SP3. A target for future optimization.

• gary

On Feb 23, 2015, at 11:47 AM, Sol Simpson notifications@github.com wrote:

I had thought this was a bug with how sp3 reports the mouse location, but when I used VLC to record the sp3 screen directly the mouse was ok.

How do we know that VLC is not just capturing the actual mouse cursor graphic as part of the overall screen capture?

Confero View and the Screen Capture Filter (correct?) rely on Windows API to get the mouse position, and then draw mouse cursor graphic. So does this not suggest that there is a difference somewhere between what the reported screen resolution is vs. what resolution bounds mouse position is being reported in by the S3?

— Reply to this email directly or view it on GitHub.

[isolver]

On Mon, Feb 23, 2015 at 12:00 PM, garyfeng notifications@github.com wrote:

Certainly true with Confero View's mouse cursor (the red dot). But for the streamed video, I thought it was set up so that FFMPEG gets frames from ScreenCaptureRecorder (a directDraw filter) with the mouse cursor drawn into the frame, see the addMouse() function at https://github.com/rdp/screen-capture-recorder-to-video-windows-free/search?utf8=%E2%9C%93&q=addmouse

Maybe we are saying the same thing, I'm not sure. ;) My understanding; interpretation:

The SCreenCaptureRecorder source you linked to shows that it draws the mouse cursor over top of the screen capture:

• asks the OS what the current mouse position is for each video frame: GetCursorPos
• converts this screen based value to a window based coord given the windows position: ScreenToClient
• draws mouse graphic onto screen capture image: DrawIcon

So both Confero View and the SCreenCaptureRecorder draw a mouse 'graphic' ontop of a screen image based on Windows telling the program about mouse location (using very different Windows APIs though). VLC may not do this at all, explaining why VLC does not have the issue but Confero and SCreenCaptureRecorder do

A general note: looking at the SCreenCaptureRecorder source; it draws the mouse cursor based on the mouse position at the time that the screen capture filter is basically postprocessing the screen image prior to sending it out as a video frame. There could be 10's of msec delay between when the screen capture image was taken and when the mouse overlay code is called. Confero View uses the blinking box synchronization to know when each frame was taken in the same time base that mouse events are timestamped with. So it should be more accurate in terms of selecting the mouse position for when the video frame time was.

[garyfeng]

You are right. Did the manual FFMPEG screen capture using

C:\Confero\ConferoTrack\bin\ffmpeg\bin>ffmpeg -f dshow -i video="screen-capture-recorder" output.mov

and moved the mouse to the four corners. The output (can't upload here) shows that the mouse cursor moves only 2/3 the ways for the right/bottom corners in the video.

Will find out whether it's a SP3-specific problem and whether there is any solution.

[isolver]

Try adding that line which prints out what opengl says the screen resolution is and see if it matches what the Surface 3 says it is.

My guess is that it may have the same issue as my SP1 did, where it was reporting a 1920x1080 screen resolution in monitor settings, but opengl was saying the resolution was below that. I think I fixed the issue by updating the Intel Graphics Chipset driver to the latest version.

On Mon, Feb 23, 2015 at 4:29 PM, garyfeng notifications@github.com wrote:

You are right. Did the manual FFMPEG screen capture using

C:\Confero\ConferoTrack\bin\ffmpeg\bin>ffmpeg -f dshow -i video="screen-capture-recorder" output.mov

and moved the mouse to the four corners. The output (can't upload here) shows that the mouse cursor moves only 2/3 the ways for the right/bottom corners in the video.

Will find out whether it's a SP3-specific problem and whether there is any solution.

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/101#issuecomment-75637996.

[garyfeng]

I don't have VC set up to recomple ScreenCaptureRecorder. Do you?

Or do you know of another directDraw filter that may be immune to this problem?

[garyfeng]

This may be relevant, but no solution http://stackoverflow.com/questions/26941982/touch-screen-mouse-click-positioning-with-the-microsoft-surface-3-with-windows-8

[isolver]

I do not think recompiling the filter will help. The calls are being made to dynamically linked DLLs, so it is the version of the windows libs at runtime that will matter.

If this is in regard to printing out the display resolution; I was refering to doing it from the Confero Track software. I sent instructions on how to try that a few days ago. If you want me to resend them I can.

Thanks Gary,

Sol

On Mon, Feb 23, 2015 at 4:36 PM, garyfeng notifications@github.com wrote:

I don't have VC set up to recomple ScreenCaptureRecorder. Do you?

Or do you know of another directDraw filter that may be immune to this problem?

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/101#issuecomment-75639612.

[isolver]

Interesting. When you are testing, are you using the touch screen or an external mouse? The guy in the post says that when he connects an external mouse via USB it works fine. If you can test the same, it would be good to know what the outcome is.

On Mon, Feb 23, 2015 at 4:37 PM, garyfeng notifications@github.com wrote:

This may be relevant, but no solution http://stackoverflow.com/questions/26941982/touch-screen-mouse-click-positioning-with-the-microsoft-surface-3-with-windows-8

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/101#issuecomment-75639809.

[garyfeng]

1). Using an external mouse does not change anything.

2). Added the print command. Result is [2160, 1440]

[garyfeng]

Could it be that Microsoft decides to automatically scale pix in W8.x?

see https://msdn.microsoft.com/en-us/library/windows/apps/windows.graphics.display.displayinformation.resolutionscale

and

https://msdn.microsoft.com/en-us/library/windows/apps/hh465362.aspx

[isolver]

Since openGL is reporting the same screen resolution as Windows is; that rules out the issue being the same as my SP1 had.

Re: Scaling: So you are thinking that Windows is scaling the whole desktop to, say 1920x1080, even though the screen resolution is 2160, 1440 and is reported as such? If so, based on the event data you get, is mouse location being reported in native pixel resolution but the screen capture video width * height is actually the scaled size and not 2160x1440, or visa versa?

I guess the Q's are:

1) What is the pixel w,h of screen capture frames saved? 2160x1440 or some other value?

2) Does eye position data show the same issue as mouse data?

3) Can you determine what the scaling factor is for mouse (and maybe eye) events?

[garyfeng]

Here's my reconstruction.

• Windows 8.1 on SP3 somehow reports the screen resolution as 1440x960 (scaled by 0.75), at least if you via some API.
• This is also the metric that Windows 8.1 reports the mouse position, at least the API that ScreenCaptureRecorder and ioHub use.
• ioHub thinks the screen resolution is 1440x960, with center at [720,480].
• ioHub gets the mouse position in the 2160x1440 range (??)
• When it comes to the directShow screen capture, the pixel count is still [2160x1440].
• In ScreenCaptureRecorder, the mouse position is read in the 1440x960 metrics, and is plotted on the 2160x1440 video frame, hence shrunken.
• In the Confero webapp, mouse cursor is calculated as [X-720, Y+480], where XY came from ioHub's reading based on 2160x1440. Therefore the gaze cursor moves faster than real mouse.

To your questions:

1). The saved video is 2160x1440 2). Haven't tested 3). Seem to be 1.5

[isolver]

There are some confusing things here, so it is hard for me know know what the best course of action to suggest is. Can we have a call sometime early next week to review this stuff? Since I do not have an S3, it is not possible for me to debug and I'm confused about screen resolution and mouse coord space that iohub is giving you because of some [seemingly] conflicting statements.

It seems like you are finding that iohub knows the screen resolution is 2160x1440:

@garyfeng 2 days ago: 2). Added the print command. Result is [2160, 1440]

And that the mouse coord range is as expected:

@garyfeng yesterday: ioHub gets the mouse position in the 2160x1440 range (??)

So it seems like iohub is correctly matching up based on the above. But the weird thing is that you state in the last post:

ioHub thinks the screen resolution is 1440x960, with center at [720,480].

Given what iohub printed, it knows the screen is 2160x1440. Perhaps the real issue (for Confero View), is that an incorrect offset is being applied based on some javascript reading or something, resulting in the:

, with center at [720,480].

part. Center should be 0,0 in the raw iohub data.

So based on the above, it seems like this issue should be fixable, for iohub at least, if we can determine where in the Confero View or Track code, the [720,480] extra offset is coming from.

Does not sound like anything can be done in terms of fixing the ScreenCaptureRecorder filter though. Maybe a bug should be raised with that project about this issue; the author may have a work around.

[garyfeng]

mystery solved: it turns out that by default SP3 is set to scale by 150% so that texts on the 2160x1440 screen will read like they are on the 1440x960 screen. This DPI adjustment turns out affects all kinds of things, see https://iteachstats.wordpress.com/tag/open-source-screencasting-from-ms-surface-pro/

After I disable the scaling (back to 100%), then all problems are solved. The captured mouse is correct in the saved video, and the Confero mouse position is also correct in the webapp. The only problem is that the text is infinitesimally small on the SP3.

On a side note, scaling the streaming video to 25% really cuts down the delay. Transmitting the 2160x1440 video stream drives up the CPU and makes the webapp crawling. With 0.25 scaling the delay is well less than 0.5 sec and very adequate for real-time monitoring. The CPU is at 60% or less.