Forward all mouse events from CEF to JCEF

[GoogleCodeExporter]

Hi, 
I would like to add mouse events in the JCEF browser but I cannot find any way 
to listen the event. I looked over the RenderHandler which seems to have one 
but I am confused to how can i customize my own RenderHandler in the browser.

Thanks

Original issue reported on code.google.com by ngmm...@gmail.com on 18 Nov 2014 at 9:02

[GoogleCodeExporter]

Hi ngmm,
your question is twice. At the one hand something for the JCEF-Forum 
(http://www.magpcss.org/ceforum/viewforum.php?f=17)  and on the other hand 
there is a small issue regarding this.

To get mouse events you can do what you'll do every time on java:
Component test = cefBrowser.getUIComponent();
test.addMouseListener(new MouseListener() {
    // implement the methods you need
});
test.addMouseMotionListener(new MouseMotionListener() {
    // implement the methods you need
});

Unfortunately the only events you'll currently get (in windowed rendering) are 
"mouseEntered", "mouseExited" and "mouseMoved". Neither "mousePressed", 
"mouseReleased", "mouseClicked", "mouseDragged" nor "mouseWheelMoved" events.
I think this is an issue which might be fixed soon or later.

@marshall: maybe you can change the title of this issue to "Forward all mouse 
events from CEF to JCEF" or something similar.

Regards, Kai

Original comment by k...@censhare.de on 19 Nov 2014 at 7:52

[GoogleCodeExporter]

Can these issue be fixed? Because i need it so badly I did know about the mouse 
events are not avaialble here so I used some ohter alternative by using 
Keyboard actions, but for this wheel/mouse drag it can be big hurdle for my 
current project.

Original comment by ngmm...@gmail.com on 19 Nov 2014 at 10:38

[GoogleCodeExporter]

I would be glad to use any patch/nightly builds for this.

Original comment by ngmm...@gmail.com on 19 Nov 2014 at 10:48

[GoogleCodeExporter]

I would be glad to use any patch/nightly builds for this.

Original comment by ngmm...@gmail.com on 19 Nov 2014 at 10:49

[GoogleCodeExporter]

@ngmmail: What is your use case? What exactly do you want to do with the mouse 
events?

Original comment by magreenb...@gmail.com on 19 Nov 2014 at 3:46

• Changed title: Forward all mouse events from CEF to JCEF

[GoogleCodeExporter]

I am creating a website robot it actually works on creating macro jobs. We make 
records of actions within browser and execute them independently in number of 
instances.

Original comment by ngmm...@gmail.com on 19 Nov 2014 at 3:55

[GoogleCodeExporter]

@#6: You can use off-screen rendering which already gives you complete control 
over mouse events. You will need to modify the Java code for your use case.

Original comment by magreenb...@gmail.com on 19 Nov 2014 at 4:10

[GoogleCodeExporter]

Wow it did worked but for this i need to place my .dll in my user dir 

Original comment by ngmm...@gmail.com on 19 Nov 2014 at 5:28

[GoogleCodeExporter]

If I use OSR then the part of event queue gets out of sync. 

Original comment by ngmm...@gmail.com on 19 Nov 2014 at 7:06

[GoogleCodeExporter]

Attached you'll find a patch file which solves the MouseEvents for Mac OS X.
I'm currently working on a Windows solution.

@Marshall: Could you be so kind to have a look on my patch, please?

Thanks, Kai

Original comment by k...@censhare.de on 21 Nov 2014 at 1:24

Attachments:

• 2014-11-21_get_all_mouse_events_on_mac.patch

[GoogleCodeExporter]

I am not sure where to add this patch :( 

Original comment by ngmm...@gmail.com on 24 Nov 2014 at 4:27

[GoogleCodeExporter]

#10: It looks like your patch calls mouse event selectors on the NSView created 
by CEF internally to host the windowed browser. How does this result in the 
JCEF client application receiving those mouse events?

Original comment by magreenb...@gmail.com on 24 Nov 2014 at 10:38

[GoogleCodeExporter]

@#10: Also, some comments:

1. Line 39:

+    g_browser_views_lock_.Lock();
+    for(std::set<CefWindowHandle>::iterator iter = g_browser_views_.begin();
+        iter != g_browser_views_.end(); ++iter) {

It's probably better to only hold the lock while making a copy of 
|g_browser_views_|. You can then iterate the copy without holding the lock 
(avoids problems if the selectors do something that modifies 
|g_browser_views_|, etc).

2. Line 50:

+      bool handled = false;
+      switch([evt type]) {
+        case NSLeftMouseDown:
+        case NSOtherMouseDown:
+        case NSRightMouseDown:
+          [browser mouseDown:evt];
+          handled = true;
+          break;

If the bounds match the browser window for the current iteration is there some 
reason why we want to continue iterating the for loop? Maybe we can remove the 
|handled| variable and always break at the end.

Original comment by magreenb...@gmail.com on 24 Nov 2014 at 10:46

[GoogleCodeExporter]

@#11:
Hi ngmm,
the Patch-File isn't a binary patch but a SVN-Patch file which you can use on 
the following way:

svn checkout http://javachromiumembedded.googlecode.com/svn/trunk/ your-jcef
cd your-jcef/
svn patch your/downloaded/diff/file.diff

Regards, Kai

Original comment by k...@censhare.de on 25 Nov 2014 at 9:14

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi Marshall,
Here is a new Patch-File according the mouse Events. I've added support for 
Windows as well.

@#12: Yes you're right. The solution works because the CEF frame forwards the 
mouse events to its parent.  But in fact this only works until CEF won't change 
its internal implementation. So now the mouse events are send to the parent of 
the CEF frame and therefore to Java (by its JNI magic).

@#13.1: I'm not sure if coping a std::set is that less in costs because its 
linear (O(n)) as well. But anyway, the code at this position has changed and 
therefore I'm now working on a copy of the std::set. In the windows solution I 
don't copy the std::set because in this case accessing it is less than O(n).

@#13.2: The boolean variable |handled| was only required for the lock 
mechanism. Due the implementation has changed, the boolean variable doesn't 
exist any more.

Regards,
Kai

Original comment by k...@censhare.de on 25 Nov 2014 at 9:49

Attachments:

• 2014-11-25_get_all_mouse_events.diff

[GoogleCodeExporter]

Removed my patch file because I've found an issue with it. I'll update a new 
one soon.

Original comment by k...@censhare.de on 25 Nov 2014 at 10:34

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

@#18:
Here is the final patch file according comment #17:

Original comment by k...@censhare.de on 25 Nov 2014 at 11:30

Attachments:

• 2014-11-25_get_all_mouse_events.diff

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi, 
Thanks for the patch file but I am not sure what to do after patch. I am using 
win64 platform , I ran few compile.bat, run.bat and there were some exception. 

Please help

Thanks

Original comment by ngmm...@gmail.com on 5 Dec 2014 at 7:28

[GoogleCodeExporter]

@#20: Thanks for the updated patch. Some comments:

1. Line 391:
+  // Forward params to the CefWindowHandle of Java-Canva
+  g_mouse_modifiers_ |= addModifier;
+  PostMessage(GetParent(browser), wParam, g_mouse_modifiers_ | ctrl | shift, 
pos);
+  g_mouse_modifiers_ &= ~removeModifier;

|g_mouse_modifiers_| is only accessed from MouseProc so it should probably be a 
local variable (static or not). If it needs to be static and/or global can you 
explain why?

2. We're not going to build JCEF targeting Windows versions older than XP so 
there's no need to conditionally compile based on _WIN32_WINNT values <= 0x0501.

3. Please add a stub implementation of AddCefBrowser/RemoveCefBrowser for Linux 
or add platform #ifdef's around the calls to these methods.

4. Instead of adding more non-POD globals (see  
http://google-styleguide.googlecode.com/svn/trunk/cppguide.html#Static_and_Globa
l_Variables for why this is bad) perhaps it's time to introduce a singleton 
global context object that can own these globals as members. The singleton's 
lifespan could be controlled by CefApp.cpp (created before CefInitialize and 
destroyed after CefShutdown).

5. Line 54:
+// Called by LifeSpanHandler::OnAfterCreated() to do some platform dependend
+// tasks for the browser reference like registering mouse events.

Typo -- should be 'dependent' instead of 'depended'.

6. Lines 123 and 390:
+  // Forward params to the CefWindowHandle of Java-Canva

Typo -- missing an 's' at the end.

Original comment by magreenb...@gmail.com on 5 Jan 2015 at 10:46

[GoogleCodeExporter]

I am still not sure how to use scroll events. I tried many times :(

Original comment by ngmm...@gmail.com on 5 Jan 2015 at 2:30

[GoogleCodeExporter]

@#23: Hi Marshall, thanks for your review.

> 1. You're right, a global is too much in this case. I've removed the global 
variable and replaced it by a static local one. A static variable is required 
to be able to detect mouse drag events. You have to store the modifier 
"mouse-down" and add it to each "mouse-move" event until the user releases the 
mouse button. Otherwise Java won't fire 
MouseMotionLister.moseDragged(MouseEvent e) events.

> 2. I've removed all conditions.
> 3. Added util_linux.cpp
> 5. & 6. Fixed the typos

> 4. That's true. Because we have several places in the code with globals, I 
think it would be better to apply this patch and create a new QA-issue on this 
board to address the non-POD thing there.

You'll find the updated patch file attached to this comment.

Regards,
Kai

Original comment by k...@censhare.de on 12 Jan 2015 at 9:25

Attachments:

• 2015-01-12_get_all_mouse_events.diff

[GoogleCodeExporter]

@#25: Thanks for the updated patch.

It appears that mouse pointer change no longer works correctly with this patch 
applied (testing with the detailed sample app on Windows 8.1 64-bit with a 
64-bit build of JCEF and Java version 1.7.0u67). The mouse pointer flickers 
when moving the mouse over a link but stays as the arrow pointer instead or 
changing to the hand pointer. Maybe Java is doing something with the new mouse 
notifications that causes the pointer state to be lost?

I've filed new issue #145 for the non-POD globals cleanup.

Original comment by magreenb...@gmail.com on 12 Jan 2015 at 10:17

• Changed state: Accepted

[GoogleCodeExporter]

@26: Hi Marshall, thanks for figuring that out.

On Windows 7 the mouse cursor is flickering as well.
I'm currently working on an improvement of the mouse-event solution for Windows.

Regards, Kai

Original comment by k...@censhare.de on 16 Jan 2015 at 6:33

[GoogleCodeExporter]

Still I am working on the part of mouse scroll by some alternatives method 

Original comment by ngmm...@gmail.com on 16 Jan 2015 at 6:45

[GoogleCodeExporter]

@#24 & @#28:
Hi ngmm...

if you want to use this patch you have to build JCEF on your own as explained 
here: https://code.google.com/p/javachromiumembedded/wiki/BranchesAndBuilding.

If your build environment is set up and building works, do the following:
0) Download the ".patch"-File
1) Navigate (via terminal) to your working directory (where you've checked out 
the svn trunk)
2) Optional: update your working directory (svn up)
3) apply patch: (svn patch C:/the/path/to/the/downloaded/file.patch)
4) update your work directory (gclient runhooks)
5) Compile native code and java code.

If you want to do something with the mouse events, add some mouse listeners in 
Java (e.g. to MainFrame.java):
Component c = browser_.getUIComponent();
c.addMouseListener(new MouseListener() {
  @Override
  public void mouseReleased(MouseEvent e) {
    // TODO Do something with the event
  }
  [...]  

Regards,
Kai

Original comment by k...@censhare.de on 19 Jan 2015 at 12:26

[GoogleCodeExporter]

@#26:
Hi Marshall,

the flickering mouse pointer is a delicate problem which is caused by using 
windows "PostMessage" function.
It seems to me that the native code (of the JRE) does some strange things if it 
receives mouse events.

Because I didn't found a way to trick around Java's native behavior, I've 
completely replaced the mouse event implementation for windows.
The new solution forwards all relevant native mouse events (move, pressed, 
released, horizontal mouse wheel) to the java world by calling the new 
CEF-handler CefWindowHandler.
The CEF handler itself takes the values and forwards them to the CefBrowserWr 
implementation.
The CefBrowserWr implementation creates the corresponding java mouse events and 
posts them into Java's message loop.
On this way the mouse events could be caught by implementing the corresponding 
mouse listeners as expected without influencing the native system.

And btw: the new CEF handler (CefWindowHandler) fixes issue 111, too.

Please see attached patch file, for my solution.

Regards,
Kai

Original comment by k...@censhare.de on 19 Jan 2015 at 12:27

Attachments:

• 2015-01-19_get_all_mouse_events.diff

[GoogleCodeExporter]

Hi Marshall,
I've found a small issue with the patch from comment #30 on Mac OS X which I've 
fixed in the patch attached to this comment.

The comment #30 is still valid.

Regards,
Kai

Original comment by k...@censhare.de on 29 Jan 2015 at 9:48

Attachments:

• 2015-01-29_get_all_mouse_events.diff

[GoogleCodeExporter]

Issue 111 has been merged into this issue.

Original comment by magreenb...@gmail.com on 11 Feb 2015 at 6:32

[GoogleCodeExporter]

@#31: Thanks for the updated patch. I had one minor compile error with Visual 
Studio 2013:

Line 1047:
+  std::pair<CefWindowHandle,CefRefPtr<CefBrowser> > pair =
+      std::make_pair<CefWindowHandle,CefRefPtr<CefBrowser> >(handle, browser);

1>native\util_win.cpp(231): error C2664: 'std::pair<HWND,CefRefPtr<CefBrowser>> 
std::make_pair<HWND,CefRefPtr<CefBrowser>>(_Ty1 &&,_Ty2 &&)' : cannot convert 
argument 1 from 'HWND' to 'HWND &&'

Changing this code to the following fixes the error:

  std::pair<CefWindowHandle,CefRefPtr<CefBrowser> > pair =
      std::make_pair(handle, browser);

Unfortunately the mouse cursor problem described in #26 is still occurring for 
me with the updated patch (Windows 8.1 64-bit with a 64-bit build of JCEF and 
Java version 1.7.0u67). What Windows OS and JRE version are you testing with?

Original comment by magreenb...@gmail.com on 11 Feb 2015 at 6:49

[GoogleCodeExporter]

@#33: Hi Marshall,
This issue seems to get an endless story ;-)

I've fixed that VS2013 build-problem as described by you.

Beside that I've changed the Java-code in that way, that the mouse event's 
aren't published to the AWT event-queue any more. Instead of this they're 
directly posted to the AWT component_ of CefBrowserWr.

This has the benefit that the flickering mouse curser doesn't appear any more.

Regards,
Kai

Original comment by k...@censhare.de on 13 Feb 2015 at 6:56

Attachments:

• 2015-02-13_get_all_mouse_events.diff

[GoogleCodeExporter]

@#34: Thanks, that one works as expected :). Added in revision 115 with minor 
include header ordering fixes.

Original comment by magreenb...@gmail.com on 18 Feb 2015 at 6:00

• Changed state: Fixed