search

dsaidgovsg / terraform-modules

Reusable Terraform modules
Apache License 2.0
79 stars 23 forks source link

chore(fluentd): redirect logs from hashi services #257

Closed qbiqing closed 4 years ago

qbiqing commented 4 years ago

Change td-agent config to account for that the log aggregator server may not be registered on Consul, but is a user-determined host address.

qbiqing commented 4 years ago

The newly provisioned Consul servers generated a large amount of repeating logs serf: attempting reconnect to <server ID> <IP> which I noticed while monitoring logs. Had to manually restart Consul on each instance as suggested by the issue here: https://github.com/hashicorp/consul/issues/3361 for the log to stop.

qbiqing commented 4 years ago

The newly provisioned Consul servers generated a large amount of repeating logs serf: attempting reconnect to <server ID> <IP> which I noticed while monitoring logs. Had to manually restart Consul on each instance as suggested by the issue

Actually it is unclear that this worked because the repeated logs still appear after restart (I ran systemctl restart consul in the instances). Need to investigate further.

guangie88 commented 4 years ago

This is a breaking change since it requires the derivative user to now provide a yml override like:

---
server:
  name: fluentd-server
  host: fluentd.xxx.yyy
  port: 4224