Causing browser crashes?

[diracdeltas]

Update: Several people have said that turning off SSL Observatory fixes this.

We are seeing more frequent browser crashes for some users. It seems to happen on OS X, Windows, and Linux.

This seems to be happening only on 3.4.5+. It occurs on at least Firefox 29 and 30, haven't seen it in Tor Browser yet.

Also being tracked here:

• https://trac.torproject.org/projects/tor/ticket/11700
• https://bugzilla.mozilla.org/show_bug.cgi?id=999434

Typical crash signatures:

• Linux: https://crash-stats.mozilla.com/report/index/7f9b693b-ee47-42fb-8db6-616622140617
• OS X: https://crash-stats.mozilla.com/report/index/ca4ee3ff-6549-4227-9706-0e6382140616
• Windows: https://crash-stats.mozilla.com/report/index/db43d540-bf8b-4315-aada-a96412140418

My vague impression is that it's occuring more often for Windows users.

[jackgu1988]

Same thing happens on Linux. Downgrading to version 3.4.5 and disabling the automatic updates solves the problem.

Someone should inspect the code changes on versions 3.5 and 3.5.1 and spot the issue.

[jcberthon]

The update to version 3.5.1 occured for me almost at the same time as the update of Firefox to version 29. And this occured on Ubuntu, OS X and Windows 7 at the same time. Now my wife and I are experiencing regular crashes (several times a day) of Firefox, which I attribute first to the browser. But since 3 days I have disabled HTTPS Everywhere on my Windows 7 machine and I did not experienced a single crash. I am going to continue the experience on OS X and Ubuntu both for my wife and I. Would be good that this is fixed soon.

[elias6]

I would like to say that I have been experiencing crashes in Firefox on both my 5-year-old MacBook Pro and my brand new MacBook Pro, both of which are running OS X 10.9.3 (Mavericks). I am using Firefox 30, but I remember seeing the crashes in Firefox 29 also. I can't remember if I saw it in older Firefox versions.

Firefox crashes, seemingly at random, every few hours. I think it is being caused by HTTPS Everywhere, because I disabled it and Firefox went for several days without crashing a single time.

I was using version 3.5.1 during most of the crashes. I tried upgrading to the 4.0 development version, hoping that would fix the problem, but it didn't seem to do anything.

I don't know what exactly is causing the crashes, but I have submitted some crash reports to Mozilla. Here are some of them, in case they may be helpful to anyone:

https://crash-stats.mozilla.com/report/index/ca4ee3ff-6549-4227-9706-0e6382140616 https://crash-stats.mozilla.com/report/index/69c15c3e-4807-46ae-bc07-27e502140616 https://crash-stats.mozilla.com/report/index/a7355377-60f1-4c4b-8cca-0be562140611 https://crash-stats.mozilla.com/report/index/2fd5457b-4feb-42af-be4f-0bf962140610 https://crash-stats.mozilla.com/report/index/2eea7d1b-3d63-4768-8483-4a3f92140609 https://crash-stats.mozilla.com/report/index/4c49c3c8-db81-4d3e-9efc-a7dca2140605

[diracdeltas]

Thanks, I'll look into this today. Has anyone tried version <3.5.1 on FF 29 or 30?

[jackgu1988]

Using version 3.4.5 on Linux (Firefox 30) and everything is fine here. I have disabled the automatic updates. The add-on was crashing the browser since version 28. I downgraded a while ago (version 29) and still no crashes at all.

[diracdeltas]

Current guess is that crashes were caused somewhere in the upgrade from 3.4.5 to 3.5

[diracdeltas]

But it would be good to confirm/deny whether this is happening on 3.5, since the difference between 3.5 and 3.5.1 is very minimal.

[jackgu1988]

I think it is happening in 3.5 as well. Let me use it for a day or so and I will report back if it crashes.

[jackgu1988]

OK, it only took a minute to test. Upgraded to 3.5, restarted the browser, the restored tabs started loading and Firefox crashed. I think it has never crashed again since I downgrated to 3.4.5.

[diracdeltas]

Do you have an example of a page that causes crashes?

[diracdeltas]

Actually, I think I got a crash right now while visiting https://crash-stats.mozilla.com/report/index/28fe3a51-e3ae-46e7-bcec-9e1272140616, but it didn't happen on 4.0development.17.

[diracdeltas]

For reference, this is also being discussed at https://bugzilla.mozilla.org/show_bug.cgi?id=999434 and https://trac.torproject.org/projects/tor/ticket/11700. UGH, so many bug trackers!

[andoruB]

Ah, my bad, I didn't think of looking for other similar threads. Could somebody merge the threads? I'm pretty much sure this has been occurring since FF28.0.1, but I might be wrong, but I'm very sure 29 was indeed affected. Maybe it's OS dependant? I'm currently on a Debian-based distro called CrunchBang. Unfortunately I don't know what exact pages cause the crash because I cannot reproduce the crash on the same pages, it just seems to happen randomly.

[diracdeltas]

@andoruB I got a very similar crash signature to yours just now on 64-bit debian. https://crash-stats.mozilla.com/report/index/7f9b693b-ee47-42fb-8db6-616622140617

[jackgu1988]

I do not think that it is OS related. It happens on Linux all the time. I do not think that it is website related either. It has occurred to me on all types of websites. Can you provide us with a diff of version 3.4.5 and 3.5? Maybe we can help.

[diracdeltas]

@jackgu1988 https://bug999434.bugzilla.mozilla.org/attachment.cgi?id=8441689

[jackgu1988]

@diracdeltas thanks! I compared the exact changes on github and I noticed that there have been quite a lot changes on the SSL observatory. I disabled it and no crash so far. I might be wrong, but usually it crashes after restarting the browser and having multiple tabs loading at the same time, using version 3.5.1.

I will continue testing tomorrow.

[trishankkarthik]

@diracdeltas Thanks for looking into this. If you look at my previous comment, it seems that 3.5.1, 4.0development.16 and 4.0development17 are all involved. I have found Firefox crash signatures in which HTTPS Everywhere (in either one of these versions) is involved in all reports.

[jackgu1988]

@dachshund are you using the SSL observatory?

[trishankkarthik]

@jackgu1988 Yes, but I'm not sure whether that's the reason.

[jackgu1988]

@dachshund can you try disabling it and testing whether it crashes? It has not crashed on my computer so far.

[diracdeltas]

@dachshund Yes, I saw your comments. Thank you for commenting on every possible bug tracker/mailing list. ;)

@jackgu1988 Thanks, I'll ask on the other threads whether people have SSL Obs. enabled. (I do but am not getting crashes on 3.5.1 the majority of the time)

[trishankkarthik]

@diracdeltas You're welcome :P

[Veeshush]

Guy that started https://trac.torproject.org/projects/tor/ticket/11700 here.

Yep, I too have had SSL Observatory enabled on all the machines (3 separate Win7 64bit boxes) I've had the crashes on. I'm using https-everywhere-4.0development.17.xpi at the moment and still get crashes. I haven't tried downgrading to 3.4.5 yet.

edit I haven't tried disabling SSL Observatory yet either. But I think that could be very related to the crashes though, cause again, all 3 of the systems I did have it enabled.

[andoruB]

@jackgu1988: When I said OS-specific I meant that maybe the problems with HTTPS Everywhere popped out during FF29.0 while on other OSes might have started much earlier, like in my case (FF 28.0.1)

I'll try and see if I get any crashes from now on with the SSL observatory disabled.

[jackgu1988]

@Veeshush since it crashes even when there are no websites loading, it seems logical to be the the observatory. Anyway, no crashes yet on 3.5.1 since I disabled it.

@andoruB sorry, I was referring to the first post of the page, where it says "My vague impression is that it's occuring more often for Windows users.".

[Argon-]

Count me in as an affected OSX user then. It happened over multiple versions of Firefox (I deactivated it finally in 29 but it started at least in 28).

For me it always happened when I clicked on a tab, especially when this triggered loading the website for this tab. I can't say for sure whether the actual loading-of-a-site process is needed or if it could happen for about:blank too. At first I suspected a bug in the current FF version but since it didn't stop with 29 I started to deactivate extensions one by one. The crash was frequent enough for this to test and soon HTTPS Everywhere was identified as culprit. Unfortunately my crash reports differed every time drastically, so I was never able to deduce some useful information from them.

[jackgu1988]

Well, no crashes so far with the observatory feature disabled. Under different circumstances it would have crashed more than 2-3 times by now.

[diracdeltas]

@jackgu1988 I received another report by email that disabling Obs. seemed to fixed it. Person mentioned there was a reddit Firefox thread where multiple people came to the same conclusion.

So yay, that narrows it down.

[Argon-]

I gave it a try too (OSX, Firefox 30, Extension 3.5.1, deactivated observatory) and can confirm it so far.

[jackgu1988]

@diracdeltas that's great! I can take a look at the code on Friday, maybe I will spot something if it is not fixed until then. Thanks!

[diracdeltas]

@jackgu1988 Cool! Peter and I were going to have a hackathon to fix this and do a release on Friday (Pacific time), perhaps we can sync up over IRC. I would look at git diff refs/tags/3.5 3.4.5 -- src/components/ssl-observatory.js

[jackgu1988]

@diracdeltas yes, this file inspired me in order to come to the conclusion that it might be the problem. I will try to join if I have time, but unfortunately my real-life job is a priority.

[jgillula]

I'm trying to help out with tracking this down, but I can't seem to replicate the crash. (I've been using 3.5.1 since it came out on up-to-date versions of FF on 64-bit Linux Mint 13 and have never experienced a crash.)

If anyone who experiences the crashes regularly under Linux (maybe OSX too) is willing to help, you could do the following:

• Enter about:config in the location bar and click "I'll be careful, I promise!"
• In the "Search:" field that shows up near the top of the page, enter extensions.https_everywhere.LogLevel
• Double-click on the entry in the "Value" column, enter 1 and hit OK
• Close your browser, then open up a command-line terminal
• Run firefox by pasting the following command into the terminal and hitting enter: firefox | grep 'SSL Observatory' > tmp.log && tail tmp.log > observatory.log && rm tmp.log

This should create a log of the last ten debug messages from the SSL Observatory in the file observatory.log. If someone could post that file, we might be able to see where in the SSL Observatory code Firefox is crashing...

Note that this log may contain information about what URLs you've been browsing--feel free to scrub those out if you like first. Alternatively, if you're not comfortable posting the log publicly, you can always contact me directly to get it to me.

[jackgu1988]

@jgillula here is my log http://ur1.ca/hk4m6

Does not seem very helpful as it is not pbs.twimg.com specific. I can reproduce it on almost any site and it does not happen every time. Sometimes it passes through.

[jackgu1988]

@jgillula looking at the code, after the final message, the submitChainArray() function gets called and it has changed indeed since version 3.4.5. I am looking at it now. Somebody else please look at it as well because I do not have much time left.

[jackgu1988]

So, I am guessing it is has to do with the warning argument that has been introduced to the function. If not, the bug should be elsewhere (I did not go deep though).

[jackgu1988]

Final update and leaving for today:

The bug may be in the buildRequest() function which is called via the submitChainArray() function and is affected by the warning parameter here:

if (warning) { reqParams.push("browser_warning=1"); } else { reqParams.push("browser_warning=0"); }

The content length of the HTTP header gets created in the buildRequest() function, so it could be that something tricks the browser when attempting to forward the packet, thus it crashes. It also makes sense to be a bug of such nature. If a simple js bug was capable of crashing the whole browser, bigger problems would emerge.

Maybe it is a good idea to report upstream when we figure it out.

[jgillula]

@jackgu1988 Thanks Jack! I looked through the code myself, and I dunno how it would be the warning argument--it's explicitly set to either "true" or "false" just before all the possible places submitCertChainForChannel() (which calls submitChainArray()) is called.

You may be right--this log may not be that useful in determining where the crash actually occurs, because after logging "Cert chain is NOT whitelisted. Proceeding with submission.":

• it should've called submitChainArray()
• which should have called shouldSubmit()
• which given the earlier lines in the log, should have printed out "Already submitted cert for pbs.twimg.com. Ignoring" since the cert had already been submitted...but we never got that line in the log

If you can try running and logging it again a couple more times, maybe we'll be able to glean some more info from whether or not it always crashes in the same place?

[trishankkarthik]

@diracdeltas I studied git diff refs/tags/3.4.5 3.5 -- src/components/ssl-observatory.js and have not found an obvious suspect.

I am not convinced that SSL observatory is the culprit, because I want to know what explains the crashes. For example, why do we see crash signatures that seem to involve the incremental GC? Sometimes else the crashes seem to be emanating from the HTTP layer, so why is the problem specifically in the Observatory and not elsewhere?

As for my own anecdotal evidence, I am running 3.5.1 on Firefox Aurora 32.0a2 with SSL Observatory turned on and no crashes (yet).

Let me know how I can help with the hackathon; I should certainly like to get to the bottom of this mystery!

[jackgu1988]

@jgillula you got me wrong. I do not think that warning itself causes the crash. It is the point where we should start looking at, because it is the new thing in this function. So, following this concept, we can see that browser_warning is being added in reqParams, depending on warning, which is new.

Now, we follow browser_warning and see that it leads to the buildRequest() function. Since following the rest of the code we can see that if it passes that function we would get another log message very soon, which we did not, the bug should lie in this function. Since this function creates an HTTP header, it might be possible that something is malformed in the HTTP packet and Firefox crashes when trying to submit it. If it was just a plain simple js bug, I do not think that it would be possible to crash the whole browser.

Moreover, I managed to get the log more than once before submitting it here. 90% of the time it looks like the one I posted (with different websites most of the time).

Finally, @dachshund came to the same conclusion as I did. Please read my few last posts.

[jackgu1988]

Another thing: it might as well be destined on the server side. It might make sense to check if the addon handles well the server response.

[jackgu1988]

Also, a quick way to reproduce the crash:

1. Make sure that the observatory is enabled
2. Open many tabs (~10), preferably heavyweight, with lots of dynamic content, like Gmail, Facebook, Twitter, Tweetdeck etc
3. Close Firefox
4. Open it again and go to Menu -> History -> Restore Previous Session
5. Click on the tabs so that they will start loading
6. Most probably it will crash. If not, do the same again.

[trishankkarthik]

@jackgu1988 Yeah, it used to be that everytime I opened Gmail with Firefox 29-30, it was like playing Russian roulette: I winced and hoped that Firefox wouldn't crash. The conditioning became unbearable to the point that I had to stop using Firefox for a while.

Fortunately, no such crash has happened with Aurora 32.0a2 despite tempting fate by repeatedly logging in and out in completely new browser sessions...

[elias6]

I don't have any example of a page that causes the crashes or any behavior at all other than just having the extension installed and enabled. When I say Firefox seems to crash at random, I mean exactly that. After Firefox crashes, I usually restart it and pick the option to reopen the tabs I had before, and it works fine for a few hours before it crashes again.

So far, I haven't tried using the extension without the SSL Observatory.

[jackgu1988]

@mikez302 disabling the observatory will fix the issue for now.

[jsha]

I was able to repro using @jackgu1988's technique, and found this at the end of the output:

^G[29672] ###!!! ABORT: Aborting on channel error.: file /build/buildd/firefox-30.0+build1/ipc/glue/MessageChannel.cpp, line 1522
[29672] ###!!! ABORT: Aborting on channel error.: file /build/buildd/firefox-30.0+build1/ipc/glue/MessageChannel.cpp, line 1522

Line 1522 of MessageChannel.cpp, assuming it hasn't changed (http://dxr.mozilla.org/mozilla-central/source/ipc/glue/MessageChannel.cpp#), is mMonitor->AssertCurrentThreadOwns();

[jgillula]

I think it has changed, since the actual message is a little lower (http://dxr.mozilla.org/mozilla-central/source/ipc/glue/MessageChannel.cpp#1532), implying the channel isn't closing...though I haven't looked through the Mozilla code enough to know what that means...investigating more...

[jsha]

On pure speculation, could this occur if the TCP connection is closed during an attempted fetch? observatory.eff.org does seem to go through spikes of non-responsiveness that manifest as connection timeouts from the client side.

[trishankkarthik]

@jsha Maybe, but what about seemingly crashing during incremental GC?