Swarm logs service

[andywer]

Logs, stats & charts for docker containers & swarm. Super easy to set up in 5 mins.

Build a simple log collection service with a simple, but convenient UI. Make forwarding logs from a docker swarm super easy by providing an image similar to logzio-docker.

Features

• 5 mins setup
• Show logs, search them (freetext, by container, by time, ...)
• Collect docker stats and display dashboard (containers, CPU, memory)
• Free tier with decent resource limits (2GB per day, 7 days log retention, keep dashboard stats for longer)
• Minimal footprint in swarm
• Secure - Read-only access to the docker socket
• Pipe logs of any process via command line tool (nice for now.sh deployments, use pidusage for stats)

Nice to have

• App logs: Recognize JSON logger messages and parse them
• App logs: Recognize debug messages (by debug package) and parse them
• Auto-throttle data sent to service when approaching limit
• Swarm: Up-time stats
• Swarm: Show last container restarts
• App logs: Ring buffer if log level detectable
• Live tail
• Email notifications
• Slack notifications
• Alerts

[andywer]

Basic architecture

• Docker container captures data and sends it to service
• Docker container has small in-memory ring buffer, submitting only high-prio-log-level messages and their surrounding messages
• Postgres DB for long-term (stats) & mid-term (logs with prio log level) data storage
• Save stats per 10s interval or so, incl. min, max, avg, periodically do housekeeping:
  • All 10s-interval stats that are older than 3m: Aggregate into 60s interval stats
  • All 60s-interval stats that are older than 30m: Aggregate into 5m interval stats
  • All 5m-interval stats that are older than 3h: Aggregate into 20m interval stats
  • All 20m-interval stats that are older than 72h: Aggregate into 1h interval stats

Probably not even necessary

• Build a ring buffer with Redis (using LPUSH & LTRIM) where all log messages go in, but only high-prio-log-level messages and their surrounding low-level-prio messages come out

Instead:

• Make collection (log proxy) service running in client‘s cluster have an in-memory ring buffer
  • Might need to be able to temporarily disable the ring buffer / send low-prio messages as well for live tailing

Log server

• Use https://www.npmjs.com/package/graygelf or similar
• Client‘s first message must be an authentication message

[andywer]

Storing logs

In postgres. Use the default jsonb_ops when indexing the additional message fields with a GIN index. See:

• https://about.gitlab.com/2016/03/18/fast-search-using-postgresql-trigram-indexes/
• https://hashrocket.com/blog/posts/exploring-postgres-gin-index

Stats Aggregation

Edit: Just use influxdb.

See https://docs.influxdata.com/influxdb/v1.6/guides/downsampling_and_retention/.

Not an option for logs, though, due to lack of free text search.