CEF3: IO Thread crash on a URLRequest leak when shutting down cef

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Launch cefclient
2. Close the window BEFORE the page is full loaded

What is the expected output? What do you see instead?
Clean exit expected, instead there's a crash on the main process IO thread. I 
think that an cef3's internal reference to the loading URLRequest or 
URLRequestContext instance is not released when we are calling 
CefBrowserHost::CloseBrowser (and CefShutdown).

What version of the product are you using? On what operating system?
cef3 1453.1255 / Win7 64b (SDK version: cef_binary_3.1453.1255_windows)

Please provide any additional information below.
The crash is on "AssertNoURLRequests".
This error happens both in cefclient and in my app, in debug and release 
builds. Do not happens when using '--single-process' flag.

Stack trace:
libcef.dll!base::debug::BreakDebugger()  Ligne 107  C++
libcef.dll!logging::LogMessage::~LogMessage()  Ligne 650    C++
libcef.dll!net::URLRequestContext::AssertNoURLRequests()  Ligne 118 C++
libcef.dll!net::URLRequestContext::~URLRequestContext()  Ligne 44   C++
libcef.dll!net::URLRequestContext::`scalar deleting destructor'()  + 0x16 
octets  C++
libcef.dll!base::DefaultDeleter<net::URLRequestContext>::operator()(net::URLRequ
estContext * ptr)  Ligne 138 + 0x22 octets  C++
libcef.dll!base::internal::scoped_ptr_impl<net::URLRequestContext,base::DefaultD
eleter<net::URLRequestContext> 
>::~scoped_ptr_impl<net::URLRequestContext,base::DefaultDeleter<net::URLRequestC
ontext> >()  Ligne 223  C++
libcef.dll!scoped_ptr<net::URLRequestContext,base::DefaultDeleter<net::URLReques
tContext> 
>::~scoped_ptr<net::URLRequestContext,base::DefaultDeleter<net::URLRequestContex
t> >()  + 0x16 octets   C++
libcef.dll!CefURLRequestContextGetter::~CefURLRequestContextGetter()  Ligne 72 
+ 0x42 octets   C++
libcef.dll!CefURLRequestContextGetter::`scalar deleting destructor'()  + 0x16 
octets  C++
libcef.dll!base::DeleteHelper<net::URLRequestContextGetter>::DoDelete(const 
void * object)  Ligne 39 + 0x23 octets  C++
libcef.dll!base::internal::RunnableAdapter<void (__cdecl*)(void const 
*)>::Run(const void * const & a1)  Ligne 171 + 0x18 octets  C++
libcef.dll!base::internal::InvokeHelper<0,void,base::internal::RunnableAdapter<v
oid (__cdecl*)(void const *)>,void __cdecl(void const * const 
&)>::MakeItSo(base::internal::RunnableAdapter<void (__cdecl*)(void const *)> 
runnable, const void * const & a1)  Ligne 872   C++
libcef.dll!base::internal::Invoker<1,base::internal::BindState<base::internal::R
unnableAdapter<void (__cdecl*)(void const *)>,void __cdecl(void const *),void 
__cdecl(void const *)>,void __cdecl(void const 
*)>::Run(base::internal::BindStateBase * base)  Ligne 1173 + 0x19 octets    C++
libcef.dll!base::Callback<void __cdecl(void)>::Run()  Ligne 396 + 0xe octets    C++
libcef.dll!MessageLoop::RunTask(const base::PendingTask & pending_task)  Ligne 
478 C++
libcef.dll!MessageLoop::DeferOrRunPendingTask(const base::PendingTask & 
pending_task)  Ligne 491    C++
libcef.dll!MessageLoop::DoWork()  Ligne 671 + 0xc octets    C++
libcef.dll!base::MessagePumpForIO::DoRunLoop()  Ligne 523 + 0x19 octets C++
libcef.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate 
* delegate, base::MessagePumpDispatcher * dispatcher)  Ligne 64 + 0xf octets    C++
libcef.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate)  
Ligne 48 + 0x1c octets  C++
libcef.dll!MessageLoop::RunInternal()  Ligne 433 + 0x29 octets  C++
libcef.dll!MessageLoop::RunHandler()  Ligne 407 C++
libcef.dll!base::RunLoop::Run()  Ligne 46   C++
libcef.dll!MessageLoop::Run()  Ligne 314    C++
libcef.dll!base::Thread::Run(MessageLoop * message_loop)  Ligne 153 C++
libcef.dll!content::BrowserThreadImpl::IOThreadRun(MessageLoop * message_loop)  
Ligne 163   C++
libcef.dll!content::BrowserThreadImpl::Run(MessageLoop * message_loop)  Ligne 
190 + 0xc octets    C++
libcef.dll!base::Thread::ThreadMain()  Ligne 197 + 0x16 octets  C++
libcef.dll!base::`anonymous namespace'::ThreadFunc(void * params)  Ligne 57 + 
0xe octets  C++
kernel32.dll!75bf33aa()     
[Les frames ci-dessous sont peut-être incorrects et/ou manquants, aucun 
symbole chargé pour kernel32.dll]  
ntdll.dll!774e9ef2()    
ntdll.dll!774e9ec5()

Error message:
[0731/115528:FATAL:url_request_context.cc(116)] Check failed: false. Leaked 1 
URLRequest(s). First URL: http://wpad/wpad.dat.
Backtrace:
cef_time_delta [0x1013EEB1+3136401]
cef_time_delta [0x1009EBDF+2480319]
cef_time_delta [0x10524389+7221353]
cef_time_delta [0x10523CAF+7219599]
cef_time_delta [0x10523C66+7219526]
cef_time_delta [0x0FE6A8ED+169421]
cef_time_delta [0x0FE690A4+163204]
cef_time_delta [0x0FE66F46+154662]
cef_time_delta [0x0FE66E8C+154476]
cef_time_delta [0x0FE66D66+154182]
cef_time_delta [0x1044F3BF+6348959]
cef_time_delta [0x100B2297+2559863]
cef_time_delta [0x100B2258+2559800]
cef_time_delta [0x100B2115+2559477]
cef_time_delta [0x100A9E1F+2525951]
cef_time_delta [0x100A894C+2520620]
cef_time_delta [0x100A8CE4+2521540]
cef_time_delta [0x100A9830+2524432]
cef_time_delta [0x10142D52+3152434]
cef_time_delta [0x10141112+3145202]
cef_time_delta [0x101413CC+3145900]
cef_time_delta [0x100A84F6+2519510]
cef_time_delta [0x100A82CE+2518958]
cef_time_delta [0x100FB919+2860537]
cef_time_delta [0x100A7ACB+2516907]
cef_time_delta [0x100FAB86+2857062]
cef_time_delta [0x12145FD4+36719796]
cef_time_delta [0x12146125+36720133]
cef_time_delta [0x100FADB3+2857619]
cef_time_delta [0x100BC64F+2601775]
BaseThreadInitThunk [0x75BF33AA+18]
RtlInitializeExceptionChain [0x774E9EF2+99]
RtlInitializeExceptionChain [0x774E9EC5+54]

Original issue reported on code.google.com by jf.ver...@mgdesign.fr on 31 Jul 2013 at 10:00

[GoogleCodeExporter]

I have exactly the same issue. Did you find any workarounds?

Original comment by hardi.pe...@gmail.com on 14 Aug 2013 at 12:25

[GoogleCodeExporter]

No, not at this time. I did not have time to look at Cef source code yet.

Original comment by jf.ver...@mgdesign.fr on 27 Aug 2013 at 2:17

[GoogleCodeExporter]

Original comment by magreenb...@gmail.com on 27 Aug 2013 at 3:28

• Changed title: CEF3: IO Thread crash on a URLRequest leak when shutting down cef
• Changed state: Accepted
• Added labels: CEF3

[GoogleCodeExporter]

Having the same issue

Original comment by grk...@gmail.com on 2 Sep 2013 at 2:56

[GoogleCodeExporter]

This issue seems to be harmless as it happens only when quitting application,
but it gives the impression of instability in application, so this is quite
critical in the corporate environment in which we use CEF. 

This crash can also happen even when webpage seem to be completely loaded. 
I suspect that some requests might just hang up for "forever", I've noticed 
many times such behavior in browsers, and this is the probably the case.

There was a similar issue in CEF 1, see Issue 767 and Issue 113.

Original comment by czarek.t...@gmail.com on 5 Sep 2013 at 11:12

[GoogleCodeExporter]

You could just (temporarily) disable crash notification dialogs for the time 
being, if that's actually causing you trouble in a production version of your 
application.

Original comment by hardi.pe...@gmail.com on 5 Sep 2013 at 11:17

[GoogleCodeExporter]

Is it possible to disable crash notifications only for the time when application
starts the quit procedure?

Original comment by czarek.t...@gmail.com on 5 Sep 2013 at 11:20

[GoogleCodeExporter]

I don't see why not. You could disable the notifications in the WM_CLOSE 
message which is a rather good place to place the code. You can use the WINAPI 
SetErrorMode function to achieve that.

Original comment by hardi.pe...@gmail.com on 5 Sep 2013 at 11:24

[GoogleCodeExporter]

It's harmless if cef is the last thing shutdown in your application. If you 
have some stuff behind CefShutdown (and it is the case for me), then this crash 
matters.

Original comment by jf.ver...@mgdesign.fr on 5 Sep 2013 at 11:26

[GoogleCodeExporter]

Does anyone have an example that reproduces consistently in cefclient with CEF3?

Original comment by magreenb...@gmail.com on 5 Sep 2013 at 4:45

[GoogleCodeExporter]

If you download the latest client release from the Downloads page and just run 
the cefclient.exe and immediately close it, you should get the crash 
notification. I verified it and it does happen there every time you do that.

Original comment by hardi.pe...@gmail.com on 5 Sep 2013 at 4:49

[GoogleCodeExporter]

@comment#11: Are you using the Debug or Release build? What operating system? 
Do you have any proxy settings or proxy auto-detect configured on your system?

Original comment by magreenb...@gmail.com on 5 Sep 2013 at 4:55

[GoogleCodeExporter]

I'm pretty sure it's the Release build. The last item in the list on 
http://www.magpcss.net/cef_downloads/

I am downloading it now in a Win7 64bit VirtualBox without any proxy settings 
just to confirm it once again. The first time I tested it, it was also on a 
Win7 64bit, without any proxy settings.

Original comment by hardi.pe...@gmail.com on 5 Sep 2013 at 5:09

[GoogleCodeExporter]

I could not reproduce the issue in the VBox, but on my own Win7 Home Premium 
64bit, the issue is reproducible every single time. Just have to dl the 64bit 
Release build and run the cefclient.exe and quickly close it.

If there's any way I can help you debug it, let me know.

Original comment by hardi.pe...@gmail.com on 5 Sep 2013 at 6:29

[GoogleCodeExporter]

I have reproduced this in both debug and release builds on Win7 Pro 32bit. I 
have tried VS2010 and VS2012 with the same result.

Passing --single-process resolves the problem but is not an ideal fix.

@ libcef.dll!base::debug::BreakDebugger()  Line 107

Output reads.

'cefclient.exe': Loaded 'C:\Windows\System32\dnsapi.dll', Cannot find or open 
the PDB file
'cefclient.exe': Loaded 'C:\Windows\System32\rasadhlp.dll', Cannot find or open 
the PDB file
First-chance exception at 0x7548d36f in cefclient.exe: 0x0000071A: The remote 
procedure call was cancelled.
[0915/130042:FATAL:url_request_context.cc(109)] Check failed: false. Leaked 1 
URLRequest(s). First URL: http://wpad/wpad.dat.
cefclient.exe has triggered a breakpoint

Disassembly reads as follows.

--- c:\cef3_1547_32\chromium\src\base\debug\debugger_win.cc --------------------
0F93F5F0  call        base::debug::IsDebugUISuppressed (0F9440F0h)  
0F93F5F5  test        al,al  
0F93F5F7  je          $LN5 (0F93F600h)  
0F93F5F9  push        1  
0F93F5FB  call        _exit (0FD0B824h)  
0F93F600  int         3         <<<<<<<<<<<< HERE
0F93F601  push        1  
0F93F603  call        _exit (0FD0B824h)  
$LN6:
0F93F608  int         3  

Original comment by Dave....@gmail.com on 15 Sep 2013 at 3:25

[GoogleCodeExporter]

... I reproduced this by downloading cef_binary_3.1547.1412_windows32 today and 
building it and running.

Original comment by Dave....@gmail.com on 15 Sep 2013 at 3:29

[GoogleCodeExporter]

@comment#15: Thanks, I can reproduce this problem with "Automatically detect 
settings" turned on under Internet Options. "http://wpad/wpad.dat" is the 
default URL that the proxy system attempts to load to detect proxy settings.

Original comment by magreenb...@gmail.com on 16 Sep 2013 at 3:55

[GoogleCodeExporter]

I'm not sure if --single-process fixes it. I am talking about CEF1 1384 now, 
but one of the top crashers was this assert (in both debug and release). 
I have fixed this by adding a new method (CefBrowser::CancelPendingRequests, 
and its counterpart, UIT_CancelPendingRequests) called when WM_CLOSE or 
equivalent triggers the shutdown. This will perform, as name says, cancel for 
pending the URL requests still active (including downloads), waits for count to 
drop to zero, then continues with the shutdown. Unfortunately, it meant also 
modifying Chromium URLRequestContext class (see below).

(Windows: reproduction can be enforced by triggering a WM_CLOSE send from the 
LoadStart handler, and using a page loading in longer time, such as 
nytimes.com, facebook.com - something which loads lots of elements and 
plugins). 

Schematic implementation (remember, this is CEF1):
// cef_browser.h
class CefBrowser : public virtual CefBase {
  ...
  virtual void CancelPendingRequests(bool waitForTermination) =0;
};

// browser_impl.cc
void CefBrowserImpl::CancelPendingRequests(bool waitForTermination) {
    //  this is called from UI thread; make sure it goes in IO thread
  CefThread::PostTask(CefThread::UI, FROM_HERE,
      base::Bind(&CefBrowserImpl::UIT_CancelPendingRequests, this, waitForTermination));
  return;
}
...
void CefBrowserImpl::UIT_DestroyBrowser() {
  ...
  if (request_context_proxy_.get()) {
    UIT_CancelPendingRequests(true); // ===> add this

    // Delete the proxy on the IO thread.
    CefThread::DeleteSoon(CefThread::IO, FROM_HERE,
        request_context_proxy_.release());
  }
  ...  
}

void CefBrowserImpl::UIT_CancelPendingRequests(bool waitForTermination) {
  REQUIRE_UIT();

  std::unique_ptr<base::WaitableEvent> request_destroyed_event;
  if(waitForTermination) {
      request_destroyed_event.reset(new base::WaitableEvent(false, false));
  }

  if (request_context_proxy_.get()) {
    CefThread::PostTask(CefThread::IO, FROM_HERE,
        base::Bind(&BrowserRequestContextProxy::CancelRequests, 
        base::Unretained(request_context_proxy_.get()), 
        request_destroyed_event.get())
    );

    if(request_destroyed_event.get()) {
        request_destroyed_event->Wait();
    }
  }
}

// src/net/url_request/url_request_context.h
class NET_EXPORT URLRequestContext
    : NON_EXPORTED_BASE(public base::NonThreadSafe) {
  ...
  // cancel all requests, signaling the event if passed
  void CancelRequests(base::WaitableEvent* destroyEvent);
  ...
};

// src/net/url_request/url_request_context.cc
void URLRequestContext::CancelRequests(base::WaitableEvent* destroyEvent) {
  int num_requests = url_requests_ ? url_requests_->size() : 0;
  if(num_requests != 0) {
      // we need to destroy the requests AND wait for all to go into destructor
      // if an URLRequest remains alive AND is destroyed, its __dtor will access
      // URLRequestContext which is destroyed.

      // we need to make a copy since url_requests_->erase(request) 
      // will be called from the URLRequest __dtor.
      std::set<const URLRequest*> requests = *url_requests_;
      for(std::set<const URLRequest*>::iterator itreq = requests.begin();
          itreq != requests.end();
          ++itreq) {
          URLRequest* request = const_cast<URLRequest*>(*itreq);
          if(request != 0) {
              request->Cancel();
          }
      }
  }

  if(destroyEvent) {
    destroyEvent->Signal();
  }
}

Hope this can be of some help for devising a temporary fix until the real one 
gets done.

Original comment by chaos.de...@gmail.com on 17 Sep 2013 at 6:16

[GoogleCodeExporter]

@comment#17: The AssertNoURLRequests due to pending proxy connections is fixed 
in trunk revision 1445, 1547 branch revision 1446 and 1453 branch revision 1447.

A build including this fix should be available in the next day or so on 
http://cefbuilds.com (pay attention to the revision number to make sure you 
have a build including the fix). Please test and let me know if you're still 
experiencing the assertion.

Original comment by magreenb...@gmail.com on 17 Sep 2013 at 8:20

• Changed state: Fixed

[GoogleCodeExporter]

That's good news. This will cover only pending proxy connections only, correct?

Original comment by chaos.de...@gmail.com on 17 Sep 2013 at 9:38

[GoogleCodeExporter]

@comment#20: Correct, and this fix only applies to CEF3.

Original comment by magreenb...@gmail.com on 17 Sep 2013 at 9:44

[GoogleCodeExporter]

I have found this issue in revision 1562
So is this really fix?

Original comment by justin.r...@gmail.com on 23 Jan 2014 at 9:15

[GoogleCodeExporter]

I'm also seeing a similar stack trace to this in user's crash dumps.  We're 
using CEF 3.1916.1781

Original comment by r...@yahoo-inc.com on 5 Aug 2014 at 10:05

[GoogleCodeExporter]

There are two separate issues here (linked to a some point, but it seems racy).

1. Shutdown crash #1 - AssertNoURLRequests. 
My fix have two parts (sadly, this means modifying also Chromium, not only CEF).
a. Add a new CEF function CefPreShutdown.
b. Add a global event (mine is called 
Local\<someprefix>::PreShutdownEvent.<pid> (someprefix is application specific, 
pid is the process ID loading CEF, usually main process ID).
c. CefPreShutdown is calling SetEvent on the event from b.
d. Application calls CefPreShutdown when closing (usually on WM_CLOSE).

In chromium/src/net/url_request/url_request_context.cc:
+namespace {
+   bool ShuttingDown() {
+     wchar_t event_name[256] = L"";
+     _snwprintf_s(event_name, _countof(event_name), _TRUNCATE, 
+       L"Local\\<someprefix>::PreShutdownEvent.%8.8x", 
+       ::GetCurrentProcessId());
+     HANDLE preshutdown_event = ::OpenEventW(SYNCHRONIZE, FALSE, event_name);
+     if(preshutdown_event == NULL) {
+       return false;
+     }
+     DWORD wait_result = ::WaitForSingleObject(preshutdown_event, 0);
+     ::CloseHandle(preshutdown_event);
+     if(wait_result == WAIT_OBJECT_0) {
+       return true;
+     }
+     return false;
+   }
+};

void URLRequestContext::AssertNoURLRequests() const {
  int num_requests = url_requests_->size();
  if (num_requests != 0) {
+   if(ShuttingDown()) {
+     // output debug string with what leaks
+
+     // leak the requests intentionally since we're on shutting down
+     return;
+   }
... 
}

2. Shutdown crash #2 - ResourceDispatcherHostImpl::CancelRequestsForContext
A somewhat similar crash is happening also on shutdown. Is it not easy to repro.
The crash occurs on loaders_to_cancel.clear() because it seems a race condition 
on pending_loaders_ release vs destroy.
Fix:
void ResourceDispatcherHostImpl::CancelRequestsForContext(
    ResourceContext* context) {
...
- loaders_to_cancel.clear(); 
+ if(ShuttingDown()) {
+   // leak intentionally loaders on shutdown
+   for (LoaderList::iterator i = loaders_to_cancel.begin();
+      i != loaders_to_cancel.end(); ++i) {
+      (*i).release();
+   }
+ }
+ else {
+   loaders_to_cancel.clear(); // <== this is the line crashing on shutdown
+ }
...
}

I cannot submit a patch since Chromium is also patched, but the implementation 
should be straightforward.

Notes.
- Using CEF3 r1916 codebase on Windows.
- ShuttingDown() is the same as in case #1.
- Both fixes are Win32 only and works by 
a) app closing is calling the new function CefPreShutdown, and 
b) modifies Chromium code to detect shutdown case and leak pending 
loaders/requests instead of crash application.
- Are not ideal since Chromium is patched, but at least they works. 
Crashes, although on shutdown, are not an option for us (we have a generic 
global crash handler interceptor to report dumps back to our server).

Original comment by chaos.de...@gmail.com on 6 Aug 2014 at 5:59

[GoogleCodeExporter]

Hi 
 I have a solution  and it's work  for all version off windows also  win7 64bit .

It's simple :

 before close you must destroy all chromium component but you must give to the system a time to really clean CEF3 framework  and Destroy  all component 

like this 

procedure TFormMain.SpeedButtonCloseClick(Sender: TObject);
begin
SaveAll;

if ( assigned (Chromium1) ) then  Chromium1.Destroy;
if ( assigned (Chromium2) ) then  Chromium2.Destroy;
if ( assigned (ChromiumPageWeb) ) then  ChromiumPageWeb.Destroy;

TimeClose.interval := 5000 ; // 5 secondes 
Timerclose.Enabled := true ;

end;

procedure TFormMain.TimerCloseTimer(Sender: TObject);
begin

Timerclose.Enabled := false ;

close;

end;

It s clear the CEF3 don't have time  to clean all component before receive new 
event 

So the solution is give him times ( 5 secondes ) , I discovert that when I use 
a button I use it to just destroy all browsers and after I click to close and 
all is fine 
Hop help 

Original comment by berrayahkamel on 22 Oct 2014 at 11:41

[GoogleCodeExporter]

A timed destroy will not solve the issue, just push the problem under the 
carpet.

Original comment by chaos.de...@gmail.com on 23 Oct 2014 at 5:41

[GoogleCodeExporter]

Re-opening this issue since it appears not to be fixed for some users.

Original comment by magreenb...@gmail.com on 5 Feb 2015 at 7:17

• Changed state: Accepted

[GoogleCodeExporter]

Trunk revision 2032 fixes issues related to request and context object lifespan.
- Simplify and document the relationship between the various context object 
types. See browser_context.h for a description of the new relationships.
- cefclient: Add `request-context-per-browser` command-line flag for testing 
multiple CefRequestContext instances.
- cefclient: Add a CefURLRequest example.

Original comment by magreenb...@gmail.com on 13 Feb 2015 at 11:19

• Changed state: Fixed