search

cztomczak / cefpython

Python bindings for the Chromium Embedded Framework (CEF)
Other
3.08k stars 473 forks source link

ApplicationSettings.debug option causing GUI stuttering when calling Python functions from JS #277

Closed cztomczak closed 7 years ago

cztomczak commented 7 years ago

According to https://groups.google.com/d/msg/cefpython/NJUDIm81D3A/eBvHwJprCgAJ it is the "debug" option that was causing the delays.

CEF Python writes to "debug.log" file using its own logger, however at the same time CEF is also writing to that same file. Maybe there is some conflict, something to do with lock acquired on the file, I don't know. Maybe CEF Python should use CEF log mechanism and that would fix it. It sounds like this is the cause. Here is the code in DebugLog.h:

FILE* pFile = fopen(g_logFile.c_str(), "a");
fprintf(pFile, "[CEF Python] App: %s\n", szString);
fclose(pFile);

It opens and closes the file for each single log write. It might be that CEF that also writes to this file, opens it during cef initialization, acquires some lock and closes it at shutdown.

OLD info

UPDATE:

When calling Python function from Javascript there is a stuttering of CSS animation during the time of the call. It looks like the UI thread is being locked for a moment. Reported on the Forum: https://groups.google.com/d/topic/cefpython/NJUDIm81D3A/discussion

At the end of this post there is presented a temporary solution that could be used for now, and two possible real solutions.

This looks like a GIL issue. GIL is a global interpreter lock in Python. Information found on the web:

Looks like the issue in question is only a momentary lock of the UI thread while acquiring GIL. Python <> Javascript calls are asynchronous, use IPC messaging between processes. Javascript runs in the renderer process, while Python runs in the browser process. You can execute time intensive tasks in Python using multiprocessing or similar, while not blocking the UI.

I recall a similar issue of UI locking when OnBeforePopup is being called in wxpython.py, in which a task is being posted on the UI thread to create a popup window.

Related information:

This are the steps during communication from Javascript to Python:

  1. Javascript runs in the renderer process. A C++ handler to javascript binded function is being called. An asynchronous IPC message is being sent to the browser process, using Chromium's inter-process communication.
  2. In the browser process a CEF's C++ callback OnProcessMessageReceived is being called. From this callback we are calling into Python using ExecutePythonCallback Cython function. This func was declared "with gil" which means that GIL lock is acquired automatically.

Things to consider:

  1. Is it really a GIL issue? Confirm. Print times in OnProcessMessageReceived and in ExecutePythonCallback to see how much time it took to acquire the GIL.
  2. Maybe it's a CEF/Chromium threading issue that makes acquiring GIL so much time consuming?
  3. Maybe that's just some Cython issue when acquiring GIL during a call from C/C++ code? I have hard time believing that acquiring GIL takes so much time, as presented in the video by the original reporter.
  4. How does wxPython and alike frameworks handle GIL issue? Are there some workarounds?
  5. The issue was reproduced on Windows with CEF Python 31. Does it still behave the same in latest CEF Python 54+? (Linux releases currently only available)
  6. (+) Does the issue occur also on other platforms with such long time for the UI thread being locked? (Mac, Linux)
    • (WORKS OK) Linux checked and the issue doesn't reproduce with latest CEF Python v54. Added a unit test to measure time of executing Python function and then back js callback that was passed. Time elapsed = 11 ms. Everything works immediately with no locks. Commit 6ffb865c0a95346e9343832ec4405f3591ce7ab8.
  7. Does it behave any differently in Python 3?

Temporary solution

There is an alternative for js<>py communication, from README.md:

For native communication between javascript and python use javascript bindings. Another option is to > run an internal python web server and use websockets/XMLHttpRequest for js<>python communication.

Using web server and XHR for communication wouldn't require any acquiring/releasing GIL, thus the issue should disappear I think.

Solution 1 (multi threaded message loop)

UPDATE: This improves UI locking when calling js callbacks from Python, but there is still some GUI locking when calling Python func from javascript. It seems that the final solution would need to be to use multi threaded message loop along with implementing Python-native IPC messaging with the renderer processes, without the Chromium/CEF IPC layer so that GIL doesn't need to be acquired when receiving messages (Solution 2).

If it turns out this is a Windows-only problem, then I think that there is an alternative solution, to use ApplicationSettings.multi_threaded_message_loop (Issue #133). Message loop will run in a separate thread and any calls to Python won't block UI. However whether this would also resolve issue with GIL acquiring lock taking much time, I don't know, but for sure it won't lock UI (css animation). I haven't yet played with multi threaded message loop, so don't know how this would affect current code base. There may be some issues that would need to be resolved.

Solution 2 (Python-native IPC messaging with the renderer process)

I think that it should be doable to avoid using CEF/Chromium inter-process messaging. If the issue is the GIL and we can get rid of the GIL being acquired when calling from CEF C++ function into Python in the browser process, then it should solve the UI lock issue.

We could handle IPC messaging in the browser process, directly with Python. In the renderer process we would need some cross-platform C++ code. Googled "python ipc c++" and found zeromq library that allows for sending IPC messages and can connect with any language.

cztomczak commented 7 years ago

Fixed as part of Issue #352.