Get screenshots of lots of sites when they go down

[lippytak]

It will make for a more compelling essay.

[daguar]

@Mr0grog mentioned this a while back right? Maybe he knows how to do it easily...

[Mr0grog]

Oh yeah, that was here: https://github.com/codeforamerica/project-ideas/issues/43#issuecomment-65971544 I think that service is still running…

But at the time, my thinking was: what’s the actual value there? But I guess it’s illustrative if you’re writing about the issues with these services going down. Just not really as part of a dashboard. Is that something this app should do? It should be easy enough for someone to write a separate script that calls that service when a SNAP site goes down. (Well, excepting the sites that require automated browsing, which would be a bit more complicated.)

[lippytak]

mybenefitscalwin.org

[lippytak]

Indiana

[lippytak]

Illinois

[lippytak]

Virginia https://commonhelp.virginia.gov/access/

[lippytak]

Arizona

[lippytak]

New Hampshire

[alanjosephwilliams]

Did you get one of Indiana yesterday??

On Tue, Feb 10, 2015 at 9:19 PM, Jake Solomon notifications@github.com wrote:

[image: screen shot 2015-02-10 at 9 19 17 pm] https://cloud.githubusercontent.com/assets/2533112/6142394/7f54e1ca-b16a-11e4-80ad-9a4524f769af.png New Hampshire

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/snap-it-up/issues/11#issuecomment-73835536 .

t: @alanjosephwilli p: 817 713 6264

[lippytak]

@alanjosephwilliams already here

[Mr0grog]

I added webhook code to do this automatically and save all screenshots to S3 in our ‘snap-snapshots’ bucket. However, it seems like you only get one alert group per monitor in @pingometer. Also, it seems like you have to set up each monitor individually (oof). I’ve e-mailed their support about those issues.

[pingometer]

Very cool!

You can add as many groups as your want to each monitor - there's no limit there. I replied to your ticket (re: bulk-adding a new group to all monitors).

On Mon, Feb 16, 2015 at 12:15 PM, Rob Brackett notifications@github.com wrote:

I added webhook code to do this automatically and save all screenshots to S3 in our ‘snap-snapshots’ bucket. However, it seems like you only get one alert group per monitor in @pingometer https://github.com/pingometer. Also, it seems like you have to set up each monitor individually (oof). I’ve e-mailed their support about those issues.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/snap-it-up/issues/11#issuecomment-74566103 .

[Mr0grog]

Thanks, @pingometer. Looks like this is actually working now:

https://s3-us-west-2.amazonaws.com/snap-snapshots/CA-54bc8944be653d3f86065dc5-2015-02-16T22%3A41%3A20%2B00%3A00

^ that was autogenerated from MyBenefitsCalwin going up and down this afternoon.

I wonder if we’ll always have trouble snapshotting that one since we are using Phantom… may need to look into other ways of snapshotting; maybe via saucelabs or something. :(

[Mr0grog]

(Also, you can list all the snapshots in XML by going to https://s3-us-west-2.amazonaws.com/snap-snapshots/)

[daguar]

Can Selenium with a Firefox driver be used to address that?

[Mr0grog]

Maybe; it’s just approximately five times more complicated. Don’t know how well it’ll work on Heroku, either.

[pingometer]

Can you "fake" the user agent string? If you make it appear like it's coming from Firefox or IE, it should let you through.

On Mon, Feb 16, 2015 at 2:56 PM, Rob Brackett notifications@github.com wrote:

Maybe; it’s just approximately five times more complicated. Don’t know how well it’ll work on Heroku, either.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/snap-it-up/issues/11#issuecomment-74585440 .

[Mr0grog]

@pingometer Ah! Good point, I’ll have to give that a try when it’s back up. Pretty sure you can customize the UA string in Phantom.

[pingometer]

@Mr0grog Yup, looks like it's part of page.settings: http://phantomjs.org/api/webpage/property/settings.html

[lippytak]

Reason to go deeper ^

[Mr0grog]

Also, as a diagnostic tool, I’ve got heroku scheduler now set up to run a script that will snapshot all the sites in the middle of the night. A lot of them are down during that time, so it’ll give us a nice way to get a quick look at what they all look like (and an easy thing to check to see if we’re mis-monitoring them).

[lippytak]

Louisiana has been down for a month http://dcfs.la.gov/index.cfm?md=pagebuilder&tmp=home&pid=407

[Mr0grog]

…I don’t think it’s ever been up for the entire lifetime of this project, has it?

[lippytak]

Correct. Just realized I never captured it here. But looks like it will be back up monday according to the message! (The endless monday)

On Thursday, February 26, 2015, Rob Brackett notifications@github.com wrote:

…I don’t think it’s ever been up for the entire lifetime of this project, has it?

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/snap-it-up/issues/11#issuecomment-76245444 .

[lippytak]

https://www.c4yourself.com/c4yourself/index.jsp

I know we're getting these automatically, but I'm on my comp and I just liked this one so why not.

[Mr0grog]

No, that’s good—seems like we are having some trouble screenshotting that site. Should probably see what’s going in in PageSnap… https://snap-snapshots.s3-us-west-2.amazonaws.com/CA-54a79f58be653d4687bf5eef-UP-54f3512419ad89639917e538.png

[Mr0grog]

Well, now I know what’s wrong with PageSnap, but no easy fix. Seems like there’s an SSL handshake error on C4yourself and some bug in Phantom won’t let you get around it: https://github.com/ariya/phantomjs/issues/12181

[Mr0grog]

Now generating screenshots through SlimerJS/Firefox (which doesn’t have the SSH bug Phantom does) on an extraordinarily hacked together EC2 instance (couldn’t get XVFB and Firefox running on Heroku). Same API as pagesnap, just running Firefox. Don’t really want to maintain that or make it super robust if I don’t have to. Probably better to be using a service like Browserstack or Saucelabs. Anybody know if there’s any chance we could get some non-profit help/credit from either of them? :)