Azure / fluentd-plugin-mdsd

Azure Linux monitoring agent (mdsd) output plugin for fluentd
Other
27 stars 16 forks source link

Unexpected mdsd ack status: ACK_INVALID_SOURCE logs flood #70

Open nehsharmMS opened 4 years ago

nehsharmMS commented 4 years ago

I have two sources i.e. syslog and http. and I want both of these logs to be outputted to mdsd. I tried multiple options to make it work but even if it works in some cases, there is flood of logs in /var/log/td-agent/td-agent.log :

unexpected mdsd ack status: ACK_INVALID_SOURCE 

Options tried :

  1. <match> without any tag : this works but logs flood with ACK_INVALID_SOURCE
  2. <match {gheslog.**,ifx.**}> : does not work
  3. Two mdsd plugin i.e. <match ifx.**> and <match gheslog.**>: works but logs flood with ACK_INVALID_SOURCE

Below is my fluentd conf :

<source>
type syslog
 port 25827
 bind 0.0.0.0
 protocol_type tcp
 tag gheslog
</source>

<source>
  @type http
  port 8900
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s  ## This source has tag pattern ifx.**
</source>

<match>
  @type mdsd
     djsonsocket /var/run/mdsd/default_djson.socket
     acktimeoutms 5000  # milli-seconds
    num_threads 4
      buffer_chunk_limit 2M
 buffer_type file
    buffer_path /var/log/td-agent/state/mdsd_ifx.buffer
  buffer_queue_limit 128
     flush_interval 1s
    retry_limit 10
 retry_wait 10s
</match>

PS : If I just use single <match > with any of these two tags i.e. sending only one source data to mdsd, it works perfectly without any error message.

canfikret commented 4 years ago

Please use the copy output plugin to write the input data to a file and share it with me. https://docs.fluentd.org/output/copy

<match>
  @type copy
 <store>
   @type file
    ....
 </store>
 <store>
    @type mdsd
     ... 
 </store>
</match>
nehsharmMS commented 4 years ago

@canfikret , as per our discussion and suspicion, I checked and noticed few new events started coming from syslog for which sources were not defined in mdsd.xml which must have resulted in the error ACK_INVALID_SOURCE that I had mentioned. But the problem is there are so many different events but I am not interested in all of them. Also, I don't have extensive list of all possible events from the source. But it seems if any tag that is coming from source is matched by a pattern specified in <match>, it has to be specified in mdsd.xml. Any suggestions on how can I deal with this problem?

Also, I tried <match> with multiple tags pattern on a new setup, it worked. Seems like a reliability issue.