Implement centralised logging

[lukebond]

We want all logs for all services to be tail-able together and individually, from the command line and also displayable within the UI. Searching would also be good.

From @sublimino in #19:

one-command cluster monitoring (i.e. journal -f on all units, got some fleet jiggerypokery to do this as there are silly TTY complications)

Is https://github.com/gliderlabs/logspout an option?

[lukebond]

I'd like to settle on a solution here. logspout looks good to me but not sure where best to send the logs.

@sublimino: not sure how the choice of tool (eg. logspout) relates to your above-mentioned "fleet jiggerypokery". can you outline what you have in mind?

[sublimino]

logspout -> elasticsearch-based kibana is the nicest way of doing things, my fleet jiggerypokery is to tail all the services as they start and interpolate them in the console. i.e.

$ pazctl cluster-log
[host-1] [service-a] bing
[host-2] [service-b] bang
[host-1] [service-c] bong

(this is essentially a sneaky melding of the output of fleetctl journal -f service-a fleetctl journal -f service-b fleetctl journal -f service-c)

This would be useful for cluster startup when developing, for example (much quicker feedback than going through kibana).

An extant paz cluster would probably not need that debugging as operational issues would be accessible via kibana.

^^^ This is a reflection of my bias for CLI-based tools

On 9 April 2015 at 11:27, Luke Bond notifications@github.com wrote:

I'd like to settle on a solution here. logspout looks good to me but not sure where best to send the logs.

@sublimino https://github.com/sublimino: not sure how the choice of tool (eg. logspout) relates to your above-mentioned "fleet jiggerypokery". can you outline what you have in mind?

— Reply to this email directly or view it on GitHub https://github.com/paz-sh/paz/issues/31#issuecomment-91187591.

[lukebond]

Thanks.

Sounds like that is part of the CLI and not "centralised logging". If you agree then we should create an issue in the relevant CLI repository.

Regarding the centralised solution, I'd say we have the following work to do:

• create unit files in the "paz" repo for kibana etc (is this one unit or multiple units?)
• ensure these units get installed by default alongside paz
• modify cloud config to configure log stash on each node
• add logging tab to the UI somewhere

Sound about right?

On Thursday, 9 April 2015, Andrew Martin notifications@github.com wrote:

logspout -> elasticsearch-based kibana is the nicest way of doing things, my fleet jiggerypokery is to tail all the services as they start and interpolate them in the console. i.e.

$ pazctl cluster-log
[host-1] [service-a] bing
[host-2] [service-b] bang
[host-1] [service-c] bong

(this is essentially a sneaky melding of the output of fleetctl journal -f service-a fleetctl journal -f service-b fleetctl journal -f service-c)

This would be useful for cluster startup when developing, for example (much quicker feedback than going through kibana).

An extant paz cluster would probably not need that debugging as operational issues would be accessible via kibana.

^^^ This is a reflection of my bias for CLI-based tools

On 9 April 2015 at 11:27, Luke Bond <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

I'd like to settle on a solution here. logspout looks good to me but not sure where best to send the logs.

@sublimino https://github.com/sublimino: not sure how the choice of tool (eg. logspout) relates to your above-mentioned "fleet jiggerypokery". can you outline what you have in mind?

— Reply to this email directly or view it on GitHub https://github.com/paz-sh/paz/issues/31#issuecomment-91187591.

— Reply to this email directly or view it on GitHub https://github.com/paz-sh/paz/issues/31#issuecomment-91197365.

[sublimino]

a) Agreed! b) looks good

Probably worth having individual units for clarity/reporting, but no strong feels either way.

[lukebond]

Thanks @sublimino

I'll split up into separate issues.