getlantern / browsersunbounded

Interoperable browser-based P2P proxies for censorship circumvention
GNU General Public License v3.0
14 stars 0 forks source link

Freddie memory death #197

Closed noahlevenson closed 11 months ago

noahlevenson commented 11 months ago

On October 20, 8:00 AM PST, about 24 hours had elapsed since we sent the BU link to our group of 15 beta testers.

And, at that time, I just happened to be poking the widget. I acquired 3 connections but then a lot of time passed and I acquired no more. I looked at the console and saw a bunch of errors which suggested that Freddie was not responding correctly to requests.

When I logged into Heroku, it seemed that Freddie had been reporting a ton of memory-related errors. I restarted Freddie and all returned to normal.

Freddie is sick. What's wrong with him?

image

image

image

noahlevenson commented 11 months ago

After an investigation, I don't think this has anything to do with the beta test.

Here are two trends, both of the last 24-hour period. One is concurrent censored users, and the other is Heroku resource consumption and errors.

If you overlay these trends, the camel hump lines up, which basically tells the following story:

Today, we had more censored users online than ever before. That made Freddie use more memory than he's ever used before, eventually exceeding Heroku's limits. That made him crash and emit errors. When the users began to recede, presumably due to bedtime somewhere, Freddie correctly freed his memory.

To mitigate future issues, I just upgraded Freddie to a 2X "Pro" dyno with 1GB RAM. (He had previous been hosted on a $7/month "Basic" dyno.)

Closing this out for now...

image_480

image_720