scalyr / scalyr-fluentd

The fluentd plugin for inserting log messages and metrics in Scalyr.
Apache License 2.0
6 stars 5 forks source link

Multi worker #15

Closed imron closed 4 years ago

imron commented 4 years ago

This PR adds multi-worker support to our fluentd plugin. Testing under load has shown that logs from a single docker container are always routed to the same fluentd worker.

This PR also removes the monotonically increasing timestamp restriction that was previously required by the server, and uses the tag name as the "thread_id", so that we can completely avoid the need for synchronization.

Finally, it updates the defaults to take in to account the increased maximum buffer size on the server.

Here is an example config file using 4 workers. With this config, I was able to transfer up to 20MB/s from a single Azure server based in US-East.

<system>
  workers 4
</system>

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match **> 
  @type scalyr
  api_write_token XXXXXXXXX
  use_hostname_for_serverhost true
  scalyr_server https://agent.scalyr.com/
  ssl_ca_bundle_path /etc/ssl/certs/ca-certificates.crt
  ssl_verify_peer true
  ssl_verify_depth 5
  max_request_buffer 5900000
  message_field log 
  force_message_encoding UTF-8
  replace_invalid_utf8 true
  compression_type deflate
  compression_level 3
  <buffer>
    compress text
    flush_mode interval
    flush_interval 5s
    flush_thread_count 1
    delayed_commit_timeout 30
    overflow_action throw_exception
  </buffer>
</match>
imron commented 4 years ago

Tests have not yet been updated.