Log storage/management

[JasonEtco]

Smee is great, people like it a lot and for the most part it works like a charm - but every once in a while (#63) something happens and we don't know why. Wouldn't it be great if we had log storage/management?

@bkeepers - is this something that we can add on to a Loggly account (or get a new one)? I would love to improve Smee's deployment setup, since it seems to be gaining more and more popularity.

Once we're finding money for this, I'd also like to start talking about persistence, but we can cross that bridge when we get to it.

/cc @tcbyrd

[tcbyrd]

Loggly aggregates everything by account, so we'd either have to setup a new account for smee or add it to the existing account and configure a source group (if it's a paid plan). It looks like we can easily group logs by the Heroku App name if we want to go that route. If not, I've used the free SumoLogic add-on in Heroku before for some side projects, and it worked great for that use case.

[bkeepers]

I have a loggly account for probot, but I actually just stopped using it because I kept going over the limits of the free account.

Is there a Heroku addon we could use? If not I can try to get a paid loggly account, but it'll be a lot of hassle.

[tcbyrd]

The only logging add-on that's white listed in our internal Heroku is Papertrail. I haven't used that, but it's also from Solarwinds, so at least in the same family as Loggly. I'm thinking we spin up a trial for that on Smee to evaluate if it's a good solution for our other internally hosted Probot apps?

[JasonEtco]

I'm thinking we spin up a trial for that on Smee to evaluate if it's a good solution for our other internally hosted Probot apps?

👍 :shipit:

[tcbyrd]

We're now hooked up to Papertrail on the free plan, which should be sufficient for smee at the moment. This required no changes to the app itself and we can access archives for up to 1 week if we need to. As a follow-up, I'm thinking we need to add some logging middleware to the app to get more than just "client connected" and "client disconnected" messages from the app.

[JasonEtco]

Rad. @tcbyrd - I think we should add bunyan and use our existing Probot patterns (the bunyan Express middleware, the Sentry stream, etc). I would be careful not to log all of the webhook events, I feel like they can be a lot of logs, but we can try it out and monitor it.

[tcbyrd]

I would be careful not to log all of the webhook events

👍 Yeh I don't want those in the logs. Heroku's router gives us some good info on receiving the webhooks already:

Jul 13 08:41:18 github-webhook-proxy heroku/router: at=info method=POST path="/TR341a7kbXmzw0vI" host=smee.io request_id=b0ab5d0c-a040-490e-a38d-0a523d16056e

I'm struggling to come up with what other info we need outside of that and "client connected/disconnected" at the moment, but we'll probably need it eventually.