Slimer window doesn't respond to mousewheel events

[g0morra]

Using page.sendEvent(), I've been having some trouble with some click events not registering properly on certain links, specifically on javascript links that use HTML5 attributes like data-action. To get around this, I'm using the xte utility (http://linux.die.net/man/1/xte) to send mousemoves and clicks to SlimerJS (also using Xvfb and the Fluxbox window manager, running on FreeBSD 10.0). xte uses the xorg XTEST extension to send mouse events to applications. From what I've been reading, XTEST is the most robust way to send native OS events apart from using a real mouse.

xte works very well for mouse moves and clicks with button 1. However, the way to send mousewheel data to an application on X11 is to click with button 4 (mousewheel up) and button 5 (mousewheel down). SlimerJS doesn't seem to respond to button 4 or 5 clicks via X11 running on FreeBSD.

When I leave a SlimerJS window open on my OS X dev machine, it also doesn't respond to using the wheel on my physical mouse, although it does respond to physical mouse clicks and I can use the keyboard to scroll up and down.

What do I have to change to make SlimerJS respond to simulated mousewheel input?

[laurentj]

does xte work with Firefox in your system configuration?

[g0morra]

Yes it works. Using FreeBSD 10.0 I can successfully send mousewheel scroll up / down events to Firefox with xte (by simulating a click with button 4 or button 5), using both a regular X server and Xvfb. However Slimer doesn't scroll when I send a mousewheel event in this way, although it responds fine to mousemoves and normal button 1 clicks.

On OS X you can't use Slimer with X11 because there is no firefox-x11 available as far as I know. If you use regular OS X Firefox as a base, it always shows on the main desktop and you can't get it to run inside Xvfb. However I notice that a Slimer window on the main OS X desktop will respond to left clicks and keyboard events, but it won't respond to scrolling with the mouse wheel. You can scroll up and down with the cursor keys, but not the mouse wheel.

[laurentj]

Thanks, I understand now what you want to do. I confirm that SlimerJS doesn't respond to external mousewheel events to scroll pages : I disabled scrollbars in windows. I did that because it's anoying for some features (set width/height, take screenshot etc..), and because SlimerJS is not supposed to be manipulated by an external entity (a program or a human), only by a script.

We should find a solution to respond to mousewheel events without setting scrollbars...

[laurentj]

https://bugzilla.mozilla.org/show_bug.cgi?id=400920

[g0morra]

Thank you so much! Re-adding scrollbars solved the issue. If anyone else wants to do the same thing, to add working scrollbars again, I added flex="1" and style="overflow:auto" to the browser element in webpage.xul (as described at http://www.cmiss.org/cmgui/zinc/ScrollingXulWindows), and added scrollbars=yes to the list of new window features in slLauncher.openBrowser() in slLauncher.jsm. Now Slimer windows scroll like you would expect when I send external mouse wheel events, either with a physical mouse on OS X, or by using xte to send button 4 and 5 events on FreeBSD with Xvfb.

I can definitely see the thinking behind not including scrollbars when you're not anticipating external entities to control Slimer. If your use case is something like automated screenshots then it's much better not to have scrollbars. However for my use case, the fact that the Slimer window is actually rendered via Xvfb and can be manipulated by external mouse and keyboard events is invaluable. There are some javascript links out there that for whatever reason just cannot be clicked by any synthetic mouse event that a browser generates. I looked into using sendNativeMouseEvent() in nsIDOMWindowUtils, but then you get into the same problem you have with PhantomJS - you constantly have to work out what element you're sending the mouse event to. It gets very tricky, especially if you are simulating mouse paths that mouse over and out of nested iframes. One big advantage of the way Slimer uses sendMouseEvent() is that you don't have to manually work out your target elements - but there are some links I've come across that sendMouseEvent just can't click. Much easier in those cases to use an external pointer, which means you can just let Gecko work out the mouseevents, and you can click on absolutely everything exactly as you would with a physical mouse.

Another advantage of xte is that if you're trying to simulate human mouse movements, xte lets you pause with microsecond precision when moving between pixels, so you can more accurately simulate smooth mouse movements where the mouse moves quickly or slowly at different parts of the path.

Depending on the use case, the mouse and keyboard events used by default in Slimer are definitely detectable as synthetic. For example, if you point Slimer to http://mouseflow.com/demo/, use page.sendEvent() to type some text in one of the text fields and then view the recording, you won't see any text being typed into the text field in the recording. However if you use native key events, either by using a real browser or by using Slimer and xte, you will see your text being typed in the recording.

So, needing to use external mouse and keyboard events might not be common, but it's sometimes necessary and more robust for some situations. Thanks again!

[laurentj]

Ok. Thanks for your report.

I have an idea : we could have a command line option (or a property on webpage?), saying that we want to manipulate windows with an external program like xte, and then it enable scrollbars.

[g0morra]

I think a property on webpage (maybe page.settings.scrollbars = true?) would work well, because a user would know whether or not they always wanted scrollbars for a particular script. If the setting was in their Slimer script rather than a cli arg then they wouldn't have to always remember to add that cli option every time they wanted to run that script. I use a generic cli launcher script to set up the display, the profile etc. and then launch slimer and the script I want, and I assume other people do it that way too - so the less cli args to worry about, the better.

Possibly it's better to leave the code that actually calls an external program up to the user at this point, because it means creating a child process (I use subprocess.js from the Addon SDK, which pretty much bolted straight in to webpage), and then you have to translate the document coordinates for your mouse action into screenx/screeny values and pipe them to your external program via stdin. If you want realistic mouse movement over the life time of a page view then you have to track your current mouse position as well.

[shviller]

If/when #340 gets merged, there will be a simple workaround

page.xulWindow.document.getElementById("webpage").flex=1;
page.xulWindow.document.getElementById("webpage").style.overflow="auto";

(also, it will be trivial to add this workaround as a method of webpage)

[laurentj]

@shviller this is not the good solution. It's better to have an API which allows or not scrollbars.