Host a copy of Ben's dotgov checker

[gbinal]

I think we should host our own copy of the dotgov checker by @benbalter, both to provide a more robust version of it but also because why should he have to host that?

Also, though, I think this would help spur on [some other variants of the dashboard] that we've been talking about and that folks would find useful.

The code for this is here.

cc: @konklone, @adelevie - game to help or any suggestions on how to do this?

[gbinal]

@jroo ^^^

[adelevie]

Happy to help, but will be a bit too busy for this for the next couple weeks. Also I wouldn't mind using at as a learning experience for me to get some ops/deployment chops.

On Fri, Jan 9, 2015 at 2:27 PM, Gray Brooks notifications@github.com wrote:

@jroo https://github.com/jroo ^^^

— Reply to this email directly or view it on GitHub https://github.com/18F/dashboards-on-demand/issues/2#issuecomment-69385260 .

[afeld]

This would be relatively easy to host on Cloud Foundry, which we are trying to move things to. Instructions for when I was setting up another Rails app:

https://docs.google.com/a/gsa.gov/document/d/1J81HE-XyKan8u-DKOoV5iqn2kFqJDthwqLOLEnl6JUc/edit

[adelevie]

Can we spin up a small AWS instance for this?

[konklone]

FWIW, I think we could do better on presentation than what Ben had whipped up for the gov site inspector (and probably go lighter on technology than a whole Rails app).

And I think also ideally, the data collection for all these metrics (www-redirect, HTTPS, DAP use) could be run at once, and provided over an API, and then consumed by individual dashboard(s) as needed. This is how the DAP dashboard works.

That implies some development time, though I think is worth it for the simpler devops infrastructure and the better presentation. I think this aligns several of our teammates' interests, and I'm happy to help out, especially once I get through this week.

[benbalter]

FWIW, I think we could do better on presentation than what Ben had whipped up for the gov site inspector (and probably go lighter on technology than a whole Rails app).

It's a sinatra app, not a Rails app, IIRC. Although, the datastore I used hasn't worked out well (Mongo). Theoretically, it should be mostly static data (perhaps crawled, e.g., nightly), but getting things indexed and fast proved to be a pain (at least for free).

And I think also ideally, the data collection for all these metrics (www-redirect, HTTPS, DAP use) could be run at once, and provided over an API, and then consumed by individual dashboard(s) as needed. This is how the DAP dashboard works.

The hosted gov inspector, should theoretically do just that (an API for those stats), but the challenge the browser tried to solve for (not well), was crawling, caching, and indexing.

[afeld]

Can we spin up a small AWS instance for this?

Please don't :wink: We're trying to get out of that business, and using Cloud Foundry to handle that management for us.

It's a sinatra app, not a Rails app, IIRC

Sweet, that should be even easier.

[gbinal]

Rock.

@afeld - anything I can do to help kick this off?

I think there's a number of improvements we might want to make but it'll be more intuitive to make those while looking at a live running instance.

[gbinal]

@benbalter - pardon my ignorance of heroku, but would it make sense to add us as collaborators for your hosted instance or would it make more sense for us to host our own, using Cloud Foundry.

[konklone]

OK. I think what I may start doing, then, is spinning up some standalone scanning stuff. I have some basic HTTPS scanning at https://github.com/18f/https-scan, which currently uses gov-inspector but may branch out to use some stuff from https://github.com/gutworth/httpswatch.

@gbinal We're talking about our own Cloud Foundry instance, running inside of 18F's AWS cloud.

[gbinal]

Hells yes.

[afeld]

I clarify my previous statement: if it were up to me, we should host our own copy (on Cloud Foundry) only if we want to make it 18F-specific or put 18F's name on it, but otherwise contribute to the existing project and use the existing hosting. Contribute to a 3rd-party running app just like we'd contribute to 3rd-party code.

[benbalter]

Contribute to a 3rd-party running app just like we'd contribute to 3rd-party code.

I'd love to see that code/app get some much more qualified love. Glad to add any of y'all as a collaborator, and yes, @gbinal can just as easily add anyone's heroku account so that they can deploy updates. Since it's all public data (and free hosting) there's no secret sauce to worry about hiding.

[NoahKunin]

Validating @afeld's understanding above as correct. All with the constant conditional that obligations we've already made to Federal agencies come first, and we deliver on the hours we've promised them.

[benbalter]

Just a quick update, the DotGov Browser got a big refresh, and is now a full Rails app (and works), if that helps the decision at all.