Issue with sending log files to remote server with td-agent running

[KSVarun]

Hi All,

I am trying to send log files to log aggregator and from there to s3.

Here is my source td-agent config

<source>
    @type tail
    path "/home/ubuntu/app/docker/logs/localhost_access_log.*.txt"
    pos_file "/var/log/td-agent/access-remote.pos"
    tag "remote"
    <parse>
      @type "none"
    </parse>
  </source>
  <match remote>
    @type forward
    <server>
      host <SOME_IP>
      port 24224
    </server>
    <buffer>
      @type "file"
      path "/var/log/td-agent/s3/remote-access"
      timekey 1m
      timekey_wait 1m
      chunk_limit_size 256m
    </buffer>
  </match>

Here is my aggregator config file.

<source>
    @type forward
    port 24224
  </source>
  <match remote>
    @type s3
    s3_bucket <S3_BUCKET_NAME>
    path <S3_BUCKET_PATH>
    store_as "txt"
    time_slice_format access%Y%m%d%H
    <buffer>
      @type "file"
      path "/var/log/td-agent/s3/access"
      timekey 1m
      timekey_wait 5s
      chunk_limit_size 256m
    </buffer>
  </match>

This is my log file in the source td-agent server

  2020-04-23 08:52:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.0/lib/fluent/plugin/out_forward/load_balancer.rb:55:in `select_healthy_node'
  2020-04-23 08:52:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.0/lib/fluent/plugin_helper/service_discovery/manager.rb:94:in `select_service'
  2020-04-23 08:52:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.0/lib/fluent/plugin/out_forward.rb:353:in `write'
  2020-04-23 08:52:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.0/lib/fluent/plugin/output.rb:1133:in `try_flush'
  2020-04-23 08:52:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.0/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2020-04-23 08:52:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.0/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2020-04-23 08:52:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-04-23 08:52:25 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2020-04-23 08:52:27 +0000 chunk="5a3f12bb46fdab7f4f90eb4cff17385d" error_class=Fluent::Plugin::ForwardOutput::NoNodesAvailable error="no nodes are available"
  2020-04-23 08:52:25 +0000 [warn]: #0 suppressed same stacktrace
2020-04-23 08:52:27 +0000 [warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2020-04-23 08:52:31 +0000 chunk="5a3f12bb46fdab7f4f90eb4cff17385d" error_class=Fluent::Plugin::ForwardOutput::NoNodesAvailable error="no nodes are available"
  2020-04-23 08:52:27 +0000 [warn]: #0 suppressed same stacktrace
2020-04-23 08:52:31 +0000 [warn]: #0 failed to flush the buffer. retry_time=4 next_retry_seconds=2020-04-23 08:52:39 +0000 chunk="5a3f12bb46fdab7f4f90eb4cff17385d" error_class=Fluent::Plugin::ForwardOutput::NoNodesAvailable error="no nodes are available"
  2020-04-23 08:52:31 +0000 [warn]: #0 suppressed same stacktrace

Even though the service is running on the other server, the source is saying no nodes are available

Here is my nmap output

Host is up (0.0011s latency).
PORT      STATE         SERVICE
24224/udp open|filtered unknown

Nmap done: 1 IP address (1 host up) scanned in 0.43 seconds

Telnet output

telnet <HOST> 24224
Trying <HOST>...
telnet: Unable to connect to remote host: Connection timed out

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days