images briefly broken on production

[davidmooreppf]

Hmm, maybe tweaks needed to new hosting setup? Broken-image icon was showing for all site avatar images (question targets, users).

Images were broken for a bit around 2:50 pm ET in a couple of browsers. Back now. cc @maryamgunja experienced same from a diff. IP address (we're both outside of usual OpenPlans office IP).

[walter]

Complex issue. A lot of points of potential failure:

• original image could be down (less likely if the problem is intermittent)
• the CDN may be having problems across the board (in which case all images on our site our down, but come back)
• frozen-coast is overwhelmed after a large number of requests (I had hoped adding a new web dyno to the heroku resources would help, but it hasn't much)
• heroku or AWS having problems with site hosting

It's really really aggravating since it's a relatively simple thing (image resizing and caching the images files in the CDN). I may look into alternatives to running frozen-coast. Let's talk about it soon.

[davidmooreppf]

Realize you're traveling @walter - golly, image avatars of question targets are down site-wide again. Both Twttr-added targets and from Google API.

http://www.askthem.io/us/questions/recent

random page :: http://www.askthem.io/tn/questions/recent?gov=state

http://www.askthem.io/tn/people/TNL000226

... here are the only profile image I could find that works, they're both from Twttr - the top images work, the ones on their page of q listings do not work :

http://www.askthem.io/people/55c295629d29c9323a000015

http://www.askthem.io/people/537d1f7bec8d832289000011

... I checked yesterday and images were working as expected. By the way, Airbrake payment is up-to-date, in case that affects anything.

[davidmooreppf]

Just checking the site, seems all images on production are broken again, incl. electeds from Google and all Twitter targets and user profiles, in both Chrome & Safari. Maybe they'll come back later today.

[walter]

See the embedly email from yesterday. We reached our threshold for the month.

On Oct 20, 2015, at 1:50 AM, David Moore notifications@github.com wrote:

Just checking the site, seems all images on production are broken again, incl. electeds from Google and all Twitter targets and user profiles, in both Chrome & Safari. Maybe they'll come back later today.

— Reply to this email directly or view it on GitHub.

[davidmooreppf]

My bad! That'll do it. Added card and upgraded to 50k URLs for $9/mo. Images are back now. We can keep this open for the earlier comment or close it now.

[walter]

Let's go ahead and close it. Closed issues can still be referred to later.

To be clear to future readers:

• we replaced our own custom image resizing service with embedly awhile back
• embedly has limits on its plans and we reached our threshold