ref(syslog): Use NSQ to transport log messages instead of syslog

arschles commented 8 years ago

This PR was branched from #85 and replaces #85. It uses the standard NSQ client library for Go to consume messages from NSQ, instead of another library that built an abstraction on top of the standard lib. This PR removes that abstraction in order to gain a bit more control over connection & consumption details.

Additionally, this PR represents a collaboration between @krancour and I. He is writing similar code in #94, and we're converging upon a final implementation. Asynchronous pairing?

Key features of this PR

Creates a (github.com/deis/logger/consumer).Consumer interface that abstracts the NSQ connection & consumption details, so that we can swap in other distributed async transport systems (NATS, cloud queues, Kafka, etc...) later
Makes the logger crash if the NSQ service doesn't respond to connection & handshake attempts
- Leaves it to Kubernetes to re-start the logger if/when the NSQ comes back up
Makes the logger re-connect to NSQ if NSQ crashes
- This is automatically done by the github.com/nsqio/go-nsq library's re-connection timeout. See LookupdPollInterval in the Config struct
The number of goroutines that the consumer uses is configurable by an environment variable
The logger can start consuming on NSQ before any producers have started producing. Specifically, you can do the following operations and still get logs after you're done:
- Start up a logger RC
- Start up a nsqd RC
- Wait for the logger pod(s) and nsqd pod(s) to become completely ready
- Start up the fluentd daemonset
- Produce some logs

Log output when logger re-connects to nsqd:

2016/06/21 19:38:38 ERR    1 [logs/consume] (10.171.245.26:4150) IO error - EOF
2016/06/21 19:38:38 INF    1 [logs/consume] (10.171.245.26:4150) beginning close
2016/06/21 19:38:38 INF    1 [logs/consume] (10.171.245.26:4150) readLoop exiting
2016/06/21 19:38:38 INF    1 [logs/consume] (10.171.245.26:4150) breaking out of writeLoop
2016/06/21 19:38:38 INF    1 [logs/consume] (10.171.245.26:4150) writeLoop exiting
2016/06/21 19:38:38 INF    1 [logs/consume] (10.171.245.26:4150) finished draining, cleanup exiting
2016/06/21 19:38:38 INF    1 [logs/consume] (10.171.245.26:4150) clean close complete
2016/06/21 19:38:38 WRN    1 [logs/consume] there are 0 connections left alive
2016/06/21 19:38:38 INF    1 [logs/consume] (10.171.245.26:4150) re-connecting in 1m0s
...
2016/06/21 19:39:38 INF    1 [logs/consume] (10.171.245.26:4150) connecting to nsqd

Closes https://github.com/deis/logger/pull/85

cc/ @krancour - I've opened this PR so that we can review our respective changes.

jchauncey commented 8 years ago

I love this change. Not sure why I didn't think of it.

codecov-io commented 8 years ago

Current coverage is 79.25%

Merging #90 into master will not change coverage

@@             master        #90   diff @@
==========================================
  Files             3          3          
  Lines           135        135          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits            107        107          
  Misses           15         15          
  Partials         13         13

Powered by Codecov. Last updated by 3bf21b8...12fd016

arschles commented 8 years ago

Note: this PR is going to be closed, in favor of #94. See https://github.com/deis/logger/pull/94#issuecomment-227900376 for details

krancour commented 8 years ago

This should have been closed by #94. Closing now.

deis / logger

ref(syslog): Use NSQ to transport log messages instead of syslog #90

Current coverage is 79.25%