Leave Confero View Tab and go to another chrome tab problem

[garyfeng]

1. When I open the ConferoView on Chrome as one tab, and switch to a different tab for a while (say 30 sec.), and then switch back to the ConferoView tab, the interface will freeze for a while -- almost 30 sec., before slowly coming alive.

What happens when the tab is not active? Are events still being handled? Looks like the View tab is trying catch up with a lot of data/events.

Seems to happen only when the recording is in session. When the recording is stopped, there doesn't seem to be any delays.

Of course we can tell people not to check facebook while monitoring an experiment ;-)

[isolver]

So after doing some research, this behaviour is being caused by the fact that the frame renderer is using the requestAnimationFrame API to schedule decoding and drawing of the frames. When the browser window has focus, this is the right thing to do, as it allows for canvas updates at the monitors refresh rate (assuming the processing time of each call is < the refresh rate of coarse).

The issue is that the requestAnimationFrame API is 'too smart', and browsers have built in logic that disables calls to requestAnimationFrame when the user opens another browser tab (logic being why waste cpu on doing animation calls to a tab that likely can not be seen), and then starts it up again when they go back to the animation tab.

So basically, screen capture video frames continue to be sent as long as recording is occurring, but when the webApp tab is not in front, the frames are not being processed and are instead being cued up. So when you go back to the page, it 'locks up' while the video renderer runs through the frame build up, finding the next appropriate frame to show.

Before requestAnimationFrame , animation was done by using setTimeout to schedule repeated calls to a rendering loop. But it is very inaccurate so gettings smooth animation using it for stuff like video or webgl games is not possible.

Even setTimeout is dropped to a 1 second interval when the tab is hidden, so it is not a real solution here anyhow.

Ideally, the javascript decoder and canvas renderer should be modified so that it is much faster and throwing out backlogged frames and getting to the current frame to show. How to do this I do not know right now.

So, will try to improve / fix, but may need to live with the constraint of "don't leave the webapp page when recording!" rule.

[isolver]

For now, this needs to be a doc issue.

[garyfeng]

I can confirm that this also affects the gaze and mouse cursors. But under normal normal cases we should not have this issue. I agree this is a DON"T DO THIS issue on the training side.

[garyfeng]

copying Jingyuan's description from email

Huang, Jingyuan
Sent:    Monday, June 30, 2014 8:57 AM
To: 
Sol Simpson [sol@isolver-software.com]; Feng, Gary
Hi Sol,

I just started a new experiment (1 new session 2 calibration 3  validation 4 eye overlay 5 
mouse overlay) and left lab. When I came back after 15 minutes, the eye overlay was gone; 
mouse overlay stayed on the screen but not tracing the mouse any more. I had program 
running for 1 hour and overlay never came back normal.

I restarted the windows, the eye overlay came back normal but mouse overlay stayed in 
the same position all the time.

That’s how it worked on Friday. I may have sometime this afternoon to retest it.

Best,

Jingyuan

[garyfeng]

I actually encountered the issue twice today now and have it on video. It looks like ioHub is not taking mouse events, at least not forwarded to the Confero webapp. In the first case, when I started Confero (after a successful session before), the mouse cursor shows as a red dot but it's not tracking the mouse. The Mouse Info pane shows no new events. Then I quit, and started Confero again. this time I couldn't even turn on the mouse cursor- it didn't show up. I will email you the second video.

Will follow up.

[garyfeng]

Tried again, and the mouse cursor is still not working. This time it seems to have detected an initial event at time 0, but no updates after that. I will restart the computer and test again later.

[isolver]

When leaving Confero running for 15 minutes - 1 hour; is any eye being tracked, or is the eye missing the whole time? If the eye is missing this whole time, I bet one thing that could be improved, and help how quickly things start responding again faster, has to do with the parser.

During periods of no eye data it buffers up the eye samples, but there is no limit to the buffer size! Should probably make clear the buffer every 20 seconds or so, and generate some type of error event, starting the parser over.

If the system is actually being used, this is a non issue of course, but would be good to do for general robustness I guess.

Sol

On Mon, Jun 30, 2014 at 12:19 PM, garyfeng notifications@github.com wrote:

I can confirm that this also affects the gaze and mouse cursors. But under normal normal cases we should not have this issue. I agree this is a DON"T DO THIS issue on the training side.

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/79#issuecomment-47552352.

[garyfeng]

In her 1-hour test, there weren't eyes to track for the most part, so this could be an issue. Would you please file an issue for future work to implement the buffer cleanup you mentioned above?

But my tests yesterday showed no problem with the eye cursors for 5-10 min tracking (with intermittent missing eyes); it the mouse cursor. I am going down to the lab to look into the HDF5 file to see if any mouse events are logged.

[garyfeng]

Looks like ioHub failed to get mouse events for the sessions 4-6 whichI had problems with. See attached.

[garyfeng]

can't replicate the issue. I am leaving it running without an eye for a while. Will check back.

[Update]: let it run for 15 min with locked screen and had no issues when I came back. Will have to do further testing

[garyfeng]

Meanwhile, wrt the switching to another tab problem, if you open not a new tab but a new browser window with the Webapp tab still showing, then the interface works, not surprisingly. So if you really needed to check factbook, you can. :-)

[isolver]

this is good to know. ;)

Re mouse events; if no mouse events occurred during the recording periods for a session, none would be recorded.

On Tue, Jul 1, 2014 at 10:23 AM, garyfeng notifications@github.com wrote:

Meanwhile, wrt the switching to another tab problem, if you open not a new tab but a new browser window with the Webapp tab still showing, then the interface works, not surprisingly. So if you really needed to check factbook, you can. :-)

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/79#issuecomment-47662005.

[garyfeng]

there were definitely mouse movements, which was captured on the video but not in iohub. I am assuming that this would be an ioHub-level problem, not a Confero issue. I haven't replicated this bug today, though. Will report when/if we do.

On Jul 1, 2014, at 12:06 PM, Sol Simpson notifications@github.com wrote:

this is good to know. ;)

Re mouse events; if no mouse events occurred during the recording periods for a session, none would be recorded.

On Tue, Jul 1, 2014 at 10:23 AM, garyfeng notifications@github.com wrote:

Meanwhile, wrt the switching to another tab problem, if you open not a new tab but a new browser window with the Webapp tab still showing, then the interface works, not surprisingly. So if you really needed to check factbook, you can. :-)

— Reply to this email directly or view it on GitHub https://github.com/isolver/Confero/issues/79#issuecomment-47662005.

— Reply to this email directly or view it on GitHub.

[isolver]

OK. Never seen or heard of this happening before though.

If it is happening, it must be related to the load on the tracker PC being so high that iohub is being blocked from even running. If that was the case, then there should also be no device updated seen on the webapp; the video could keep being updated, but nothing else.

[garyfeng]

I looked at the video again, and while the CPU was 90-95%, we did get gaze updates, just not mouse events, neither in HDF5 nor for the webapp.

I will let you know if we continue to have this issue.

[isolver]

On Thu, Jul 3, 2014 at 2:16 PM, garyfeng notifications@github.com wrote:

eo again, and while the CPU was 90-95%, we did get gaze updates, just not mouse events, neither in HDF5 nor for the webapp.

I will let you know if we continue to have this issue.

The only possible way this can be happening (which I have only read about, never seen before), is if the Windows low level hook used to get the mouse events is taking more than 0.5 sec to process a single mouse event. If this happens, the MSDN docs say the hook is removed from the system.

This is the only possibility, the LL hooks used get events prior to the desktop mouse even getting them (you can block mouse events from even reaching the desktop if you wanted). So if iohub is not getting them, but the desktop is, then the LL hook has been removed from the system by the OS because of it not servicing events fast enough.

The only way this would happen is if the CPU is overloaded so much can not actually even keep up with the cpu needs of the processes running. Only way to fix this is to use a faster system, or make the major process hog (ffmeg) use less CPU; perhaps by running at a lower sceen resolution, or frame rate.

Could also try raising the priority of the iohub process to high so it gets picked over ffmpeg, the eye tracker service, and the screen capture filter, more often.